Leaford's proposed new, streamlined, review system.

[leaford]

Leaford’s Refined and Streamlined Reviewing System

It’s been several months since I made my reviewing system, and I have learned a lot since then, so it seemed like time to update the system. I appreciate that many people like my current system as it is, but I’ve always believed that nothing is so perfect it can’t be improved. And more, if you don’t re-examine your own methods and beliefs from time to time, you can’t learn or grow. And if you aren’t learning or growing, all you’re doing is waiting to die.

I had a couple of things in mind to improve; first I wanted to get rid of the subcategories, and reduce the number of performance measures to 5, both to make it more concise and easy to explain, and because I found that each individual performance measure would be too diluted by being averaged then averaged again, under the original system.

Secondly, I wanted to eliminate the appearance and build quality category, and stick to performance based categories. Those two got kind of shoehorned together and were never a comfortable fit. Appearance is too subjective and doesn’t need to be rated on a 1 to 5 scale anyway. And build quality is too seldom an issue, so it can be just mentioned in the narrative review.

And third, I wanted to add cartridge life as a rated category.

I am keeping the 1 to 5 scale, with 1 being unacceptable, 2 being poor, 3 being average or good, 4 being excellent, and 5 being outstanding. And I will keep the same relative values for the categories I am keeping, so those scores will still be comparable to the older reviews.

So the new rating system looks like this:
Performance measures:

1) Vapor Production; Where a 1 is little to no vapor, 2 is thin and wispy, 3 is acceptable (most minis fall here), 4 is excellent (most penstyles fall here), and 5 is outstanding (Sedansa Diva or Janty Classic are examples). However, for a few high vapor devices, I will allow 1 or even 2 bonus points, so they can earn a 6 out of 5.

2) Ease of Draw; Where a 5 is like sipping water through a straw, 3 is like a slurpy, and a 1 is a thick milkshake.

3) Battery life; Based on 5-second inhalations, where a 1 40 inhalations or less, a 2 is 60, 3 is 80, 4 is 100, and 120 or more would be a 5. Again, for extremely high battery life, I may allow 1 or 2 bonus points.

4) Cartridge life; Also based on 5-second inhalations, where a 1 is 20, 2 is 40, 3 is 60, 4 is 80, and 5 is 100 puffs. Again, for extremely high cartridge life, I may allow 1 or 2 bonus points.

5) Reliability and ease of use; this is a broader category than I would have liked, but I needed to combine something to keep under 5 measures, and these seemed the best fit. Mostly, this will represent the switch performance, but will also include things like atomizers that need to be warmed up with a lot of primer puffs, LEDS that aren’t clearly visible to the user, noisy atomizers, and basically anything else affecting performance.

These 5 performance measures will be averaged into an Overall Performance score.

Besides rating the performance measures objectively, I also wanted to start giving a ranking score, which will of course derive first from the performance ratings, but will also allow me to include my personal, subjective opinion, and also other factors like price, customer service, company reputation, etc.

This will be a 5 tier system, where the Top Tier, or Top Shelf, is wholeheartedly recommended, the Second Tier or Shelf is recommended but with reservations or cautions. The Middle Tier is acceptable, but not really recommended (they work, but there are better for the price, as one example). The Lower Shelf is not recommended, and the Bottom Shelf is strongly recommended against.

So, a review template would look like this:

Template-----------------------------------------------------------------------
Name:
Category:
Website:
Price:
Shipping, tax, etc.
Warranty/Guarantee:


Tier Ranking:
What I Like:
The Downside:

Overall Performance:
1) Vapor Production:
2) Ease of Draw:
3) Battery Life:
4) Cartridge Life:
5) Reliability and Ease of Use:


Customer Service Experience:


Contents:
Extras:


Battery charging times:
Interchangeable with:


Narrative review:
-----------------------------------------------------------------

In addition, I am going to try to use a more uniform format for my videos; first an introduction including an unboxing, the pros and cons, and my tier ranking. Second, the body of the review, including the ratings, and last a demo portion including any comparisons to other models, any tips or tricks, maybe even accessories.

And lastly, I have taken the discussion a couple weeks ago on e-cigarette-forum about inhalation techniques to heart. It seems there are two major types of smokers, and therefore e-smokers. There are those who inhale the smoke or vapor directly down into their lungs, and those who first collect the smoke in their mouth, and then inhale. I was always a direct inhaler with analog cigarettes, and actually didn’t realize other people mouthed their smoke like a cigar or pipe, before inhaling. What’s more, a poll I conducted showed that mouth-puffers were a clear majority, by well over 2 to 1.

So, I have spent the past couple weeks practicing the mouth-puff method, and will be using it a lot more in my videos. I will try to demonstrate both inhalation methods, and include in my evaluation whether the device being reviewed works better with one method or the other. In a few cases, I think the mouth-puff method would really have changed my opinion of some devices I’ve reviewed in the past, like the Smoke 51, which performs better with a mouth-puff than a direct inhalation. Or the Sedansa Diva, which is actually TOO easy a draw for the mouth-puff method to feel comfortable, although it is ideal for direct inhaling.

And, personally, I have started to use the mouth puff method most of the time. Like I said, you learn and you grow, or else you’ve already started dying.

So, I will be filming the first of several upcoming videos in this new format tomorrow, but I wanted to put it out for some public comment first. What do you guys think? Is it still missing anything? Is the Reliability and Ease of Use performance measure too broad? Any other suggestions? Lay it on me.

[Vince1]

How about adding model number to the template, or will that be listed under the name line? (complete name and model number). I really like the interchangeability section. I submitted a suggestion for an interchangeability section in this forum for posts about what fits what, but who knows?

[BrazilyanGuy]

Why don't you include a throat hit category? For me this is the most important thing in e-cig.

[leaford]

Thanks for the input, guys!

How about adding model number to the template, or will that be listed under the name line? (complete name and model number). I really like the interchangeability section. I submitted a suggestion for an interchangeability section in this forum for posts about what fits what, but who knows?

Oops, yeah I do include that, just forgot it on the template. Thanks for pointing that out!

Why don't you include a throat hit category? For me this is the most important thing in e-cig.

I meant to discuss Throat Hit, sorry.

I have only just begun to notice throat hit, since I switched to mouth-puffing. See, when you direct inhale, the throat kick is always fierce, unless you use a very smooth e-liquid like Freesmoke, and so I never noticed much difference between different models.

Now that I'm mouth-puffing I'm noticing the subtler differences, and I will begin including that in my narrative review. But I can't give everything a 1-5 rating, and I am only beginning to evaluate throat hit, so no stats, just light, heavy, medium, etc.

[tribalmasters]

Throat hit is different for everyone but could be useful, I remember reading somewhere that Leaford direct inhales, others mouth it! I don't know if Mike enjoys a smoke on one but I am sure he does the same as his dad

-EDIT-Damn posted above me just as I was writing it hehehe!

[Terraphon]

Leaford ;

If you're going to use this as your review system, I've got to make some suggestions.

First off - Cartridge life; This is going to vary from cig to cig and cart to cart and, in my opinion, shouldn't be used as a measurable unless there's a set of controls in place. If you're going to review cartridge life you have to make sure that you treat the cartridges the exact same way, every time, with no exceptions...Allow me to quantify;

I get more puffs per cart with puresmoker juice than with JC juice. The JC juice is thicker and doesn't seem to sit in the cartridge media as well.

I also get more puffs per cart when I remove the media, clean it thoroughly and fill the empty cart 1/2 way, then put the media back in and then fill it the rest of the way, waiting for the liquid to pool and then pulling the media flush to the top of the cart.

If you're going to rate carts, you have to devise a system by which you ensure the rating is factual, objective and fair with as few potential differences between samples as possible.

Make a process...drain, clean, dry, drip, replace media, top off until the liquid start to pool and then pull the media up to soak it up...something like that.

Ease of draw; This is subjective and should not be subjected to scoring. Some people like a heavier draw, some people like a lighter draw. I love my evo, for example, and it is a MUCH harder draw than a regular cigarette. I have also noticed that draw can chage from battery to battery. On my evo, for example, I get a MUCH smoother, easier draw through my branded intellicig batteries than I do through the genie battery that has no cutoff. I can use the same cart and atomizer between the batteries and there is a DRAMATIC difference.

Rather than making this a measurable, I think it should be part of a subjective discussion. Remember, what you like may not be what other people like and since many people that see your videos will never read this discussion, a subjective view can all too easily skew an opinion.

Vapor production; Another category that is going to vary and must have some controls put in place. The amount of vapor that is produced can swing wildly from liquid to liquid. My EVO carts crank out a good amount of vapor with the factory juice in them. With Puresmoker juice in there, I get heaps more vapor. Change to JCSJ and I get vapor production somewhat in the middle. (If you see my liquid reviews, you'll see that I have a process for reviewing liquids that virtually eliminates any possibility of the results being skewed by hardware. I even have a "review only" atomizer that I never use except for when reviewing liquids (Sure I've only done 2 at this time but there will be many, MANY more to come).)

If you're going to rate vapor you must make sure that you are using the same juice for every cig and every review...Remember that you're reviewing the performance of the hardware, not the performance of the juice that comes with it (and may have been sitting in a warehouse for half a year before it got to you). If you want to review the vapor production as the product comes from the box, please include a disclaimer in each video that states that you're basing vapor off of the manufacturers carts and whatever liquid may, or may not, be in them. Make sure that people know the results may not be accurate for the hardware.

I would suggest using categories 1, 3, 4 and 5 as measured categories (with controls in place to assure 100% consistency) and going with #2 as a subjective discussion rather than a numbered rating.

Also keep in mind that the juice you use WILL effect the amount of "throat hit" you get so if you're going to do a subjective discussion on that, as well, make sure you're sticking to one particular juice.

If you choose to just review the things as they come from the manufacturer / distributor then, again, please make sure you're stating that clearly in all of your reviews and making sure that "non-users" know that the results can / will vary based on external factors.

Of you have any questions / comments, I'm all e(ye)ars.

[leaford]

Leaford ;

If you're going to use this as your review system, I've got to make some suggestions.

First off - Cartridge life; This is going to vary from cig to cig and cart to cart and, in my opinion, shouldn't be used as a measurable unless there's a set of controls in place. If you're going to review cartridge life you have to make sure that you treat the cartridges the exact same way, every time, with no exceptions...Allow me to quantify;

I get more puffs per cart with puresmoker juice than with JC juice. The JC juice is thicker and doesn't seem to sit in the cartridge media as well.

I also get more puffs per cart when I remove the media, clean it thoroughly and fill the empty cart 1/2 way, then put the media back in and then fill it the rest of the way, waiting for the liquid to pool and then pulling the media flush to the top of the cart.

If you're going to rate carts, you have to devise a system by which you ensure the rating is factual, objective and fair with as few potential differences between samples as possible.

Make a process...drain, clean, dry, drip, replace media, top off until the liquid start to pool and then pull the media up to soak it up...something like that.

Ease of draw; This is subjective and should not be subjected to scoring. Some people like a heavier draw, some people like a lighter draw. I love my evo, for example, and it is a MUCH harder draw than a regular cigarette. I have also noticed that draw can chage from battery to battery. On my evo, for example, I get a MUCH smoother, easier draw through my branded intellicig batteries than I do through the genie battery that has no cutoff. I can use the same cart and atomizer between the batteries and there is a DRAMATIC difference.

Rather than making this a measurable, I think it should be part of a subjective discussion. Remember, what you like may not be what other people like and since many people that see your videos will never read this discussion, a subjective view can all too easily skew an opinion.

Vapor production; Another category that is going to vary and must have some controls put in place. The amount of vapor that is produced can swing wildly from liquid to liquid. My EVO carts crank out a good amount of vapor with the factory juice in them. With Puresmoker juice in there, I get heaps more vapor. Change to JCSJ and I get vapor production somewhat in the middle. (If you see my liquid reviews, you'll see that I have a process for reviewing liquids that virtually eliminates any possibility of the results being skewed by hardware. I even have a "review only" atomizer that I never use except for when reviewing liquids (Sure I've only done 2 at this time but there will be many, MANY more to come).)

If you're going to rate vapor you must make sure that you are using the same juice for every cig and every review...Remember that you're reviewing the performance of the hardware, not the performance of the juice that comes with it (and may have been sitting in a warehouse for half a year before it got to you). If you want to review the vapor production as the product comes from the box, please include a disclaimer in each video that states that you're basing vapor off of the manufacturers carts and whatever liquid may, or may not, be in them. Make sure that people know the results may not be accurate for the hardware.

I would suggest using categories 1, 3, 4 and 5 as measured categories (with controls in place to assure 100% consistency) and going with #2 as a subjective discussion rather than a numbered rating.

Also keep in mind that the juice you use WILL effect the amount of "throat hit" you get so if you're going to do a subjective discussion on that, as well, make sure you're sticking to one particular juice.

If you choose to just review the things as they come from the manufacturer / distributor then, again, please make sure you're stating that clearly in all of your reviews and making sure that "non-users" know that the results can / will vary based on external factors.

Of you have any questions / comments, I'm all e(ye)ars.

Thanks for the thoughtful comments, Terraphon.

I measure the cartridge life using original cartridges from the manufacturer, not refilled, washed, etc. I am reviewing the device as it is, not modified in any way, even small ways like refilling cartridges. The question isn't how many puffs do you get from a generic homogenized cartridge, it's how many puffs do you get from the cartridge the consumer gets out of the box.

The puffs, just like for the battery life, are as close to 5 seconds each as I can manage. This is not meant to be a measurement of actual use, but a comparative measurement, using consistent 5-second units so it is directly comparable from device to device.

Things like refilling, interchangeability, etc, are good tips and tricks, and I usually include some of that, and intend to do more of it. But they will be in the final demo section. None of that should be included in the performance measurements of the device itself.

Similarly, vapor levels and throat kick should be judged using the manufacturer cartridges. I don't want to maximize the vapor levels by using a high producing liquid. Nor do I want to artifically level the playing field between them all. I play them as they lay. I judge them in the same state as the consumer will find them when they unpack their own kit.

There is one exception, the upcoming Ruyan v8. I had to change liquids because I couldn't fairly judge an invisible vapor.

As for ease of draw, while whether or not you PREFER an easy or hard draw is, indeed subjective, whether or not the draw IS hard or soft is purely objective. It stands as an objective performance measure. That's why I always provide the breakdown, not just the averaged score. If you do not like an easy draw, look for a low score in that area.

In the end, any review has strong elements of subjective preference. That's the whole point, isn't it? I like to give as much objective data as I can at the same time. But no, I'm neither as objective nor as precise as a scientist would be. I aspire to be more like Consumer Reports, but without machine testing.

[Terraphon]

Thanks for the thoughtful comments, Terraphon.

I measure the cartridge life using original cartridges from the manufacturer, not refilled, washed, etc. I am reviewing the device as it is, not modified in any way, even small ways like refilling cartridges. The question isn't how many puffs do you get from a generic homogenized cartridge, it's how many puffs do you get from the cartridge the consumer gets out of the box.

The puffs, just like for the battery life, are as close to 5 seconds each as I can manage. This is not meant to be a measurement of actual use, but a comparative measurement, using consistent 5-second units so it is directly comparable from device to device.

I can absolutely see your point, here. If you're interested in measuring the base device as it is sold to the consumer, however, I think you should just make that clear in your reviews. Here's why;

When I saw a bunch of evo reviews I decided to get one. I purchased it, it arrived, I started puffing. I was GROSSLY disappointed, though, with the amount of vapor and throat hit I was getting. I tried another cart and saw a major difference. Satisfaction started to set in. I took the first cart and cleaned it out, seeing that it only had a tiny little drop of liquid in it. I then placed some aftermarket liquid in it, swapped out again and HOLY SH.T I thought my head had caught on fire with the vapor I was getting out of it.

Due to that, I don't believe that testing on the factory carts is the most accurate representation of what the consumer CAN get out of their product. I guess I'm enough of an early adopter type (read: cynic) that I somewhat expect to have issues with anything that's factory assembled and presented to me as a complete package.

I just don't want people to ever get the wrong idea about the potential of a device because of what may be a half dry cartridge.

Things like refilling, interchangeability, etc, are good tips and tricks, and I usually include some of that, and intend to do more of it. But they will be in the final demo section. None of that should be included in the performance measurements of the device itself.

If you do this (I would actually have a drip of a good quality juice on standby so I can show, at the end, what the device CAN do) that would address any of the issues I can see with the system.

Similarly, vapor levels and throat kick should be judged using the manufacturer cartridges. I don't want to maximize the vapor levels by using a high producing liquid. Nor do I want to artifically level the playing field between them all. I play them as they lay. I judge them in the same state as the consumer will find them when they unpack their own kit.

Again, I see your point and I hope you understand mine, from above. I think that, sometimes, the test drive can be a bit misleading...in either direction.

There is one exception, the upcoming Ruyan v8. I had to change liquids because I couldn't fairly judge an invisible vapor.

I'll assume that's something to do with the liquid they use (until I see the review)

As for ease of draw, while whether or not you PREFER an easy or hard draw is, indeed subjective, whether or not the draw IS hard or soft is purely objective. It stands as an objective performance measure. That's why I always provide the breakdown, not just the averaged score. If you do not like an easy draw, look for a low score in that area.

I thought about this after I posted and the thing that creates the issue for me is the conditioned response people will have. If someone likes having to hit their e-cig like a milkshake, that's their thing...If they see the "review score" for that category as a 1, however...Pavlov, etc...

In the end, any review has strong elements of subjective preference. That's the whole point, isn't it? I like to give as much objective data as I can at the same time. But no, I'm neither as objective nor as precise as a scientist would be. I aspire to be more like Consumer Reports, but without machine testing.

Ok, this is where you and I differ. I prefer a more controlled, scientific approach with as little subjectivity or room for interpretation as possible (12 years as an engineer...what can I say?) where you're more comfortable with anecdotal findings and opinion. There's nothing wrong with that it's just a different way.

That being said, I'll have to watch some of your reviews under the new format and see how they play out. I hope you'll bear in mind some of the potential for misleading but I also believe that you'd not intentionally post a bunk review.

And now I have to figure out which liquids I want to order and review for next week. I want to do 5 or more vids but I need to choose my poisons.

We'll see how the various suppliers out there add up.

[leaford]

When I saw a bunch of evo reviews I decided to get one. I purchased it, it arrived, I started puffing. I was GROSSLY disappointed, though, with the amount of vapor and throat hit I was getting. I tried another cart and saw a major difference. Satisfaction started to set in. I took the first cart and cleaned it out, seeing that it only had a tiny little drop of liquid in it. I then placed some aftermarket liquid in it, swapped out again and HOLY SH.T I thought my head had caught on fire with the vapor I was getting out of it.

Due to that, I don't believe that testing on the factory carts is the most accurate representation of what the consumer CAN get out of their product. I guess I'm enough of an early adopter type (read: cynic) that I somewhat expect to have issues with anything that's factory assembled and presented to me as a complete package.

What? You're talking about bad cartridges. OF COURSE if the cartridges arrive half empty I'm not going to base my review on that.

That's why I don't do unboxing type reviews. I take my time to evaluate them carefully before judging how it rates, writing the review, then filming the demo. And if there are bad parts, including cartridges I exchange them, or refill or whatever before I judge the device, and certainly before I film the video.

But, I don't go beyond making sure I have good cartridges or parts. I don't maximize performance artificially because that's not a review.

See, to me, if a review used something other than factory liquid to get tons of vapor to impress you, then THAT would be deceptive, and should be disclosed. Using the manufacturer's cartridges should be the assumption, not a special disclaimer; just as I wouldn't expect to have to specify that I am using the manufacturer's battery, but damn well ought to specify if I am not.

Again, i don't think a review is about maximizing performance, that's a tip or trick. Both are valuable, but they are very different and shouldn't be mixed.

And it is where they DO get mixed, that it should be disclosed, to avoid someone being dissapointed because they can't get the same performance I did.

I just don't want people to ever get the wrong idea about the potential of a device because of what may be a half dry cartridge.

And I don't want people to get the wrong idea about what sort of performance they can expect because I am using a vapor booster.

Again, I see your point and I hope you understand mine, from above. I think that, sometimes, the test drive can be a bit misleading...in either direction.

Which is why I don't do the test drive on camera. I do the test drive over a week or so, sometimes more. The review is written based on that. The video is a demo.

I'll assume that's something to do with the liquid they use (until I see the review)

Yeah. Sorry, I should have explained, the Ruyan uses a liquid that absorbs so rapidly there is little if any visible exhalation. So I had to use regular liquid and cartridges instead. That will be disclosed, or disclaimed, when I film it, of course.

[Terraphon]

What? You're talking about bad cartridges. OF COURSE if the cartridges arrive half empty I'm not going to base my review on that.

That's why I don't do unboxing type reviews. I take my time to evaluate them carefully before judging how it rates, writing the review, then filming the demo. And if there are bad parts, including cartridges I exchange them, or refill or whatever before I judge the device, and certainly before I film the video.

My mistake, dude. I was under the impression that your review format was going to be changing to an unboxing style. I read this:

In addition, I am going to try to use a more uniform format for my videos; first an introduction including an unboxing, the pros and cons, and my tier ranking. Second, the body of the review, including the ratings, and last a demo portion including any comparisons to other models, any tips or tricks, maybe even accessories.

to mean that you would be doing it that way. I guess I should assume that the unboxing / intro / etc...will be done on different days or at different times, yes?

But, I don't go beyond making sure I have good cartridges or parts. I don't maximize performance artificially because that's not a review.

See, to me, if a review used something other than factory liquid to get tons of vapor to impress you, then THAT would be deceptive, and should be disclosed. Using the manufacturer's cartridges should be the assumption, not a special disclaimer; just as I wouldn't expect to have to specify that I am using the manufacturer's battery, but damn well ought to specify if I am not.

Again, i don't think a review is about maximizing performance, that's a tip or trick. Both are valuable, but they are very different and shouldn't be mixed.

And it is where they DO get mixed, that it should be disclosed, to avoid someone being dissapointed because they can't get the same performance I did.


And I don't want people to get the wrong idea about what sort of performance they can expect because I am using a vapor booster.


Which is why I don't do the test drive on camera. I do the test drive over a week or so, sometimes more. The review is written based on that. The video is a demo.

I completely agree with this. I wouldn't "maximize" performance, either. I was merely suggesting a leveled playing field based on a misinterpretation on my part. Again, I was under the impression that you'd be doing an unveiling straight into a review and that, IMHO is asking for disaster

If you're doing it over time then there's really no discussion, here.

Yeah. Sorry, I should have explained, the Ruyan uses a liquid that absorbs so rapidly there is little if any visible exhalation. So I had to use regular liquid and cartridges instead. That will be disclosed, or disclaimed, when I film it, of course.

Gotcha.

I don't know that I'd be satisfied with that. I need my vapor to feel complete.