Preface

Or, the time I accidentally influenced an entire industry through a survey question (click to read more)

In 2015, I created a website called fursuitreview.com (FSR). Fursuits (one of a kind whole body animal costumes) are expensive and it was hard to keep track of makers that didn't have huge followings. I didn't want to run a review page, but only two existed prior to mine. One had died completely several years before, and the other one updated so infrequently it was almost useless (it, too, eventually died after FSR took off).

The other two pages were a victim of putting too much work on themselves. The first had several questions reviewers had to answer and assign points. The moderators would average all the points for each review and give the overall review a score out of ten. This ensured consistency in the scores across all reviews. It was an amazing system but it would simply take too long to do myself (plus I'm terrible at math).

Both sites were not actually sites - they were user accounts on furaffinity.net (and sometimes tumblr or livejournal) - which required write-ins to copy a big block of text and hope they got the format right (they rarely did, which meant moderators would have to correct it).

FSR started out on furaffinity, but very quickly moved to a wordpress website. I thought very carefully on how I would keep manhours and costs down. I decided to not have any user accounts at all, nor would I bother with a numbered rating system (I instead went with a "positive / neutral / negative" system because that's really what everything boils down to anyway). I devised the least amount of questions I could in order to pry the information I wanted out of fursuit buyers when they wrote in.

This resulted in two sections: wear satisfaction, and visual satisfaction.
Each section had several sample questions to help guide the writer. "Wear satisfaction" was like, do you like how it fits you? how is the ventilation? how is the vision?
Visual satisfaction was questions about looks.

Example screenshot of the FSR "wear Satisfaction" questions from 2020+

When people submitted a review, I would read over them all and make sure everything was copacetic before publishing it. Back then (before 2020), I was a one-man operation so I read literally every single review coming in. I noticed that some writers would mention lining in some of the fursuit parts. Most parts are unlined - it's just faux fur and the backing of that will be up against a wearer's skin. Lining is unnecessary and can ultimately make the costume much hotter - but it does make everything look really nice and marginally more comfortable to wear.

After a few of these, I added "Does the item have lining?" to the list of sample questions.

Does the item have lining?

Surely nothing bad could happen from such an innocuous question!!
(upside down smiley face goes here)

I must take a break here to mention that I had no experience doing literally anything that FSR required to run. I had bought 3-4 fursuits and thought it would be nice to keep track of my opinions of them. Everything else, from coding a website to writing survey questions, I had taught myself. I had some very very basic wordpress experience from a personal roleplaying character wiki wordpress site for myself, but that was it. FSR was a learn by doing experience. And boy, was it an experience. Writing, reading, reviewing, editing, customer service... FSR had me dealing with it all, and I'm surprised it gained as much traction as it did throughout that process.

So, needless to say, I had no idea what "survey bias" was.

Weeks, possibly months, later, I was browsing twitter keeping up with FSR social stuff, when I came across a tweet from a rather well-known maker (I believe it was either beetlecat or beastcub) asking other fursuit makers why customers were suddenly asking for their fursuits to be lined. I wish I had a link to this tweet, but I no longer have a twitter account. (If you happen to find it, do link it in the comments!)

It didn't hit me at first. I watched the thread and the responses roll in. As other makers replied, the sense of dread was very slowly overtaking me. Could FSR really have that much influence? Could just a single little sentence really cause so much strife?

Yes, it did, and yes... It did.

Once I connected the dots, I edited the sample questions. No longer is it simply "Does the item have lining?"
Now it is: "If it is lined, what material was used? Does it absorb sweat appropriately? Does it make cleaning easier?" But I went through a few iterations before I got to that wording.
I believe my first edit was something like "(Note: most fursuits aren't lined)", but that didn't properly imply that you shouldn't be asking your maker for lining.

After this lining snafu, I had to go through all of my questions and determine what I was accidentally influencing, how that could change what customers ask of their makers, and if I really needed that information in the review to begin with. I also, finally, learned about survey bias, and took some time to read up a little more on how to craft survey questions.

And, of course, I had to come to grips with the fact that, yes, FSR was big. And it had influence. My little side project was a staple of the community and people counted on it to make very expensive purchases. Maker reputations and business operations began to live or die by the reviews we pulled in. FSR got to be too big and too much stress for me, so I ended up giving full ownership of the site to someone else in 2022 or 2023 (it was a long transition and I'm not sure when the public announcement was made). However, as of today (31 March 2024), fursuitreview.com is still fundamentally unchanged from how I was running it - including the review form.

Nowadays, I have nothing to do with the site. I couldn't deal with the stress and responsibility once it grew to be a community cornerstone.

Unfortunately, history may be repeating itself with the project I replaced FSR with...

A brief history of BentoVid (and RICE)

I have been editing anime music videos (AMVs) since 2001. I took a hiatus from the community from 2009ish to 2018. Despite that, I still managed to make at least one video every year. In 2018, I tried to get back into the community and realized it had almost completely changed. Discord was a thing now, and it seemed most of the activity was on there. I joined a few servers, but long story short they all weren't great. In a fit of frustration and annoyance, I did the classic move of going "Screw this, I'll do it myself!"



BentoVid (called AMV Sashimi back then) was created in September 2020. Back then, I did have high hopes. I did "plan" (finger quotes) on becoming a large community, but it was like how anyone plans on winning the lottery, you know? You don't actually expect it to happen. I thought that realistically I'd get maybe 30 members and it'd be just a chill hangout spot. But in 2021, we reached the fabled 100 members. Then 200. We broke 400 this year. And outside the very first year, I didn't actively promote. BentoVid has grown purely from word of mouth since 2021.

On the other side of Discord was an annual AMV contest related to a particular anime convention. I first joined it in 2019. It was... okay... But not great. 2020 went even worse. The contest was going through a transition period and had a coordinator that, to put it mildly, was not well-liked. The concept behind the contest (which had been running for several years - I think 2005? Possibly even earlier) was great though. It was a contest focused on peer review. All the editors that submitted to the contest were the judges and voted on the winner. It was a blind contest, so nobody knew who made what, but only people who submitted to the contest could view and vote on the videos.

The feedback was cruel. People did not hold back. Still, it was helpful, and I excelled my skills a lot by participating. The contest's discord server, however, went unmoderated and the coordinator made some very bizarre decisions. Another "Screw it, I'll do it myself" event occurred. RICE - Rewards imagined by a community of editors - was born by taking that other contest and implementing all the feedback participants had been complaining about for years. February 2021 was the first RICE.

I had never run a contest before. Just like FSR (from the preface story), this was a trial by fire.
I made at least one huge mistake every year the contest ran. But ultimately people liked RICE, and word of mouth about it (and the server) grew ever faster because of it.
I had taken efforts to specifically NOT advertise RICE outside the BentoVid discord server, but that didn't stop it from growing.

4 years of RICE survey data

Partly due to my inexperience and partly due to my history with FSR, I put out RICE feedback surveys at every opportunity. I am actually not that into data. I don't analyze this stuff and I have no particular interest in collecting data at every turn for random things. I just find feedback surveys somewhat convenient and useful for my purposes. Sorry to people who are into that! XD

Pre-RICE survey data from 2021 - 2024

I have a small survey when people submit videos to RICE. It has (mostly) the same questions every year.
Vivafringe helped me go through the data, and here are the results. (links to a google sheet)

2021 - 2023, the optional survey was on the same page as video submission questions.
In 2024, I finally realized google form sections existed and I put it on a totally different page.
I went from a 100% response rate to 58%. What a huge difference! But still inspiring to see so many people went out of their way to answer regardless.

Here's my personal takeaways:

US vs Non-US
I personally thought there were more international editors participating, but it seems to hover around 20%.

Do people keep their videos a secret?
It looks like most of the survey respondents actually do attempt to keep their videos secret from everyone. However, almost as many people admit at least one person they know who will also be in RICE knows what their video is.
I honestly thought it'd be the other way around, with more people sharing betas before RICE, so this is actually pretty cool to see.

Thoughts on blind judging
This question was multiple choice with only one answer allowed, so they had to choose which meant most to them.
Most people seem to appreciate blind judging, but don't go out of their way to keep themselves blind during the event. (~70% combined)
A large minority of people admit that guessing who made what during the event is part of the fun for them. (~25%)
One possible answer was that blind judging is never truly blind - interesting to note only one person ever selected this and it was in 2023.

Main reason for entering RICE?
Another multiple choice answer where they had to select the "main" reason.
Most people join RICE because they like the BentoVid community. However, almost as many people join because they like peer review.
I'm flattered! lol
But, in all seriousness, I really expected those results would be flipped, with peer review outranking BentoVid.

Categories
A multiple checkbox question for which categories your submitted vid belongs to, according to the editor.
I mostly only care about theme, coordinator's choice, and live-action.
Theme has submissions starting at 18% in 2021 and gradually going up to 29% in 2024. 2023 is an outlier with 39%.

Coordinator's choice is very low. 1 - 3 videos. This is mixed news for me. On one hand, I think it's great people don't feel the need to pander to me (or maybe they don't know how). On the other hand... Please pander to me! XD
Live-action is something I would like to see more of... The discord server started out as an AMV server and its audience is still mostly AMV editors. But I really want BentoVid (and RICE, by extension) to be about all fanvids and vidding (Hence the name change from AMV Sashimi to BentoVid). Long story short, live-action vids are obviously a very small minority that get submitted (5 - 9 videos each year), but they fluctuate between years.
If you like live-action fanvids and want to join an active discord server... Please join us! lol

How did you learn about RICE?
This question was a small text field people could write whatever they wanted in.
No surprise people learned about it from inside BentoVid. Next highest was word of mouth-related answers like "discord," "another discord server" or "friends".
A little interesting was when AWA or POE were specifically mentioned (two popular AMV contests), but this was only twice for each.

Post-RICE survey data

2021 and 2022 had a feedback survey, but it was just one text block that asked for comments. Very few people ever filled this out and it was not very actionable feedback. 2023 was the first post-RICE feedback survey with actual guided questions, and it's because 2023 was... quite the year.

What happened during 2023 RICE?

Remember the contest I mentioned that RICE was based on? While RICE was thriving, that contest was floundering. The other contest, which for the rest of this blog I will call "HOST," happens in September - October, while RICE happens in February - March. 2022 HOST was another transition year for them. The old coordinator ("Elder") took over the contest after so many complaints about the coordinator that succeeded them ("Junior"). As far as I know, everyone was happy with Junior's leaving, but unfortunately it wasn't pretty. We weren't kind to Junior. Bridges were burned. Elder had some great ideas on how to bring HOST back up to its former glory days. But, procrastination got the better of them, so the contest started out very poorly.

Then, during the contest, they had an extremely public and embarrassing meltdown. A number of people withdrew from the contest because of it. A totally different person ("Kidd") had to take over mid-event. The discord server went through an emergency restructure so Elder didn't have any special permissions anymore. Kidd was an absolute gem and managed to run the rest of the contest on their own very smoothly (Kidd continued through 2024 and deserves accolades). But the already negative reputation of HOST definitely turned into a dumpster fire after that. It was no surprise that RICE got touted as a good alternative. I was expecting more RICE entries than the previous year but. Boy. I was just not prepared.

2023 RICE saw 56 editors and 76 videos. The prior year was only 32 editors and 45 videos. RICE does not scale well.
Moreover, I was experiencing horrible health issues at the time and really should have delayed or cancelled RICE due to them.

Consequently, some people did not have a good experience with RICE. I thought the entire year was ruined. I made a pretty involved post-RICE survey due to it.
This survey was mostly questions with text boxes where people wrote exactly what they were feeling. This made the answers a lot more personal and detailed. I will not be sharing the raw 2023 survey data.

The complaints were about:

• Too many videos for too short a time period


• Some people were rude in the discord when discussing categories


• Some people attacked one of the people giving critique in a voice call


• I tried to enforce [very badly worded] content restrictions very late into the submission window. This had editors unnecessarily scrambling to re-edit things and ultimately ended up with multiple versions of videos in the contest [which should not have happened and that is entirely on me]

To me, the responses of the 2023 survey looked pretty dire. I immediately made changes to RICE following them.

While RICE had started out as an improved version of HOST, I also had wanted the goal to be rewards for BentoVid server regulars. That's why I really didn't want to advertise it outside the server. I also never pinged @Everyone or made a special role to get updates about it. The intent was if you were around the server, you'd know it was coming, and that was that. I wanted the good peer review and critique so we could all improve. I also wanted to see amazing videos. But what I DIDN'T want - and was (surprisingly!) NEVER concerned with - was lots of randos who didn't care about BentoVid.

I explicitly never posted about RICE on a-m-v.org (despite people asking me to), and I never mentioned RICE outside my own server until after 2022 HOST. And even then it was really only in DM or if someone else had brought it up first. I still try to not advertise RICE, but I'm not as tight-lipped as I used to be. Still, it's primarily in DM.

But anyway, I'm rambling now - the point is that RICE grew outside of the BentoVid bubble. People were joining RICE who did not care about BentoVid as a whole and I had to figure out how to handle that. My previous RICE messaging of "feedback event but also contest!" was fine for BentoVid regulars. We mostly knew what we liked and understood eachother because we hung out all the time. But for people new to RICE and/or the server, they had no idea and came in with false expectations.

My two main takeaways from the 2023 Post-RICE survey were:
1. Because of my health issues and the mass increase of participants, the Discord server went (essentially) completely unmoderated during RICE.
Because RICE (and BentoVid) is usually closely moderated, many conversations/debates went on a lot longer than they should have (because no moderators stepped in), which caused a lot of stress for participants.

2. The messaging of RICE was conflicting and led people to false expectations.
People were essentially expecting HOST but "run better."
"HOST but run better" is an over-generalization of how RICE works. It's actually quite different from HOST, but without the context of being a BentoVid regular, one wouldn't have that information.

The first would be solved simply by me being present. Myself and most my staff could not be present during 2023 (honestly I'm surprised RICE ran as well as it did without us. Speaks a lot to our community!). To deal with the second point, I decided to focus on clearing up and changing RICE's messaging.

Clearing the messaging had a few purposes:

• More clearly differentiate RICE from HOST


• Discourage non-regulars from joining without being super exclusionary about it


• Discourage overtly competitive people from joining RICE


• Encourage feedback-orientated participants


• Focus more on accessibility (as RICE already applied VPR to all entries, it made sense to extend accessibility in other ways)

I took the survey responses very seriously, and as such, rushed to make announcements of what the changes would be. I ended up announcing them the same month RICE ended - March. Proof here (that is a discord link).

You can read the initial announcement there in the BentoVid server, but I ended up changing things even more, so here's the summary of what the changes ended up being:

• Very strict content restrictions (slightly relaxed later)


• Focus on feedback event FIRST (took out all mentions of "contest" and "best" on the website, replacing them with "event" and "most-liked" )


• No cash prizes at all (previously it was a $175 pool)


• More emphasis on what exactly the server culture is like and what you can expect (basically: RICE is stressful, it's full of server regulars, prepare yourself if you're new)


• Permanent categories got permanent names (previously everything was able to be voted on and changed)


• All winners only get one award (previously there were multiple designs and names made for each award)


• We added CWs as well as VPRs into the RICE expectations


• Everything possible was outlined on the website. I literally wrote out the schedule and everything that you could expect to happen, how it all worked, etc.

The fallout from 2023 RICE continued throughout the entire year. It seemed negative feelings regarding it rolled out into other issues BentoVid was having (behind the scenes, especially in the staff channels) and overall I was really not feeling great about RICE. I was seriously considering 2024 RICE being the last one I ever ran.

2023 Post-RICE survey analysis

During 2024 RICE prep (which started in October 2023), I went through the 2023 feedback again. I asked some vague questions to random people about how they felt about 2023 RICE, and their answers (most of which were not negative at all) really had me questioning my perception of the entire thing.

I gave the 2023 post-RICE survey data to a friend of mine who used to analyze that kind of thing for a living (Vivafringe). I went through the answers and redacted personal information, summarizing answers if necessary, before giving it to him.
As part of the analysis, I asked him a bunch of questions and he looked over all the data to answer those questions. Again, I will not share the actual data here (even anonymized), but I will share the analysis he provided.
Full disclosure: Viva did participate in 2023 RICE.

Analysis of Negative Experiences

Did more people have a negative or positive time in 2023?

to answer this I didn't do any fancy analysis. I read the responses and just did a vibe check of "negative" (pretty clearly had a bad time), "neutral" (had some things they didn't like, but gave other positive feedback or just in general didn't seem like they gave a shit one way or the other), "positive" ("vars you're the best" type comments)
I think "neutral" people, if you actually asked them, would say they had a positive time, but it's hard to say for sure
anyway I rated 7/25 negative, 7/25 neutral, 11/25 positive
notably a lot of people didn't respond to this survey and the non respondants were likely positive/neutral. So I don't think you should read those numbers and assume 28% of people had a negative experience
the overwhelming complaint from basically everyone was too many videos for the time they had

- 5/6 people with "negative" feedback were returnees
- 2/6 of the people who wanted "competitive" contest had negative opinions. No real signal there I think

I said the complaints about the VCs were really bad though?

I count 4 people that complained about the vcs

my takeaway from reading this is basically:
- moderate the vcs from now on, advertise them as critique/nice/whatever (you already are doing this but just saying I agree)
- if you get 76 vids again, give more time to watch stuff
I don't know how you solve chat getting tense, that was another common complaint

Category drama?
Context: some people were very vocal about RICE needing fixed categories like typical AMV contests

I read the category stuff and broadly classified them as wanting "fixed" cats, "unfixed" cats, or "neutral"
as you might expect most people didn't give a strong opinion on it, 10/25 were neutral
5 people wanted fixed, 6 people explicitly wanted to keep things as it was now (unfixed)
if you read "neutral" as support for the status quo, which I do, I think the way you're doing it is fine. especially because a lot of the fixed cat people (3/6) had negative experiences and won't be here this year

I expected a smaller turnout for 2024 RICE. All I was really hoping for were more editors than in 2022 (which only had 32).

Editors in 2023: 56
Editors in 2024: 43 (+3 more if we count DQs)
A 17 - 23% loss, depending on how you count this.

My thoughts:
1. That's not that bad, to be honest
2. If we treat 2023 as an outlier, 2024 is an acceptable and expected amount of growth from 2022.
3. One of the main complaints in 2023 was that there were too many videos, so 2024 numbers are more desirable in this context.

Other data collected from 2023 RICE

While I was mostly concerned about the negative experiences, I thankfully had the foresight to ask some other questions about RICE that gave us some valuable data.

• Kollab vs frame.io = kollab wins by landslide


• Direction for rice: contest vs feedback = 15/22 say feedback


• Will you make an account on a website to do rice stuff? 12/25 say no


• basically everyone answered the "what does rice do differently" question with "the feedback is a lot better". So a way to make rice better is to streamline process for providing feedback.

How long people spend on RICE vids:

• 10/21 (of the people that responded) started working on their rice vid 1 month or longer in advance


• the competitive people ("A contest where the best of the best wins") seem kind of in the middle, time wise. 3/6 spent less than 1 month, 2/6 spent 2 months exactly, 1 didn't respond.


• there were 8 newcomers, but basically same conclusion. 4/8 started more than a month in advance. 3/8 took a few weeks

Google docs & PSVs
RICE offers a google doc with video information instead of trying to put it all in a filename. We also offered detailed VPRs and CWs, and, in some cases, alternate versions of videos that are more friendly for photosensitive users (called PSVs).

• 3/25 people said they used the VPRs.


• 18/25 people used the infosheet


• 4/25 used the CWs


• 3/25 used PSVs

What about 2024 Post-RICE survey data?

I collected that. This blog entry is already the size of a novel so I'll put it in another entry, I guess. Stay tuned!
(Don't hold your breath though)

I will exit this entry with the following results from that survey:

If you participated in 2023 RICE, what would you say your overall experience was?
15/18 answered overall positive
2/18 said they did not participate in 2023
1/18 said neutral
0/18 said overall negative

If you participated last year (2023), would you say your experience THIS year (2024) was:
9/17 answered overall better than last year
7/17 said about the same as last year
1/17 said did not participate last year
0/17 said overall worse than last year

Your overall 2024 RICE experience was:
17/18 answered positive
1/18 said neutral
0/18 said negative
0/18 said boring

2024 RICE had 43 editors and 62 videos.