How to properly conduct an experiment

[Matt Timion]

Experiments must be run in a certain way in order to reduce error. Error is introduced into any experiment as a result of conditions that are different from group to group. What this means that is that we must attempt to keep as many things the same as possible between experimental conditions.

I will explain the rules for participating in experiments, and then give an example.

Before testing a particular method, you must pick a stretch of highway/free which you can travel on uninterupted for a long period of time. If you can travel for 100-120 miles one way, it is better. The reason we pick highways and freeways over city driving is simple. There is too much potential error to be introducted into city driving. For example, if before testing you have all red lights, and after making a modification you have all green lights, it would be unknown if the increased gas mileage is due to the modification you made or due to the fact that you had zero idling time. Traveling on the highway reduces error that can be introduced due to idling, red lights, trains, traffic, etc.

You will first obtain a base measure of your gas mileage. To do this you must fill up your gas tank, travel the strip of highway/freeway and then travel back. You must then fill up immediately. Divid total miles driven by total gallons of gas and that is your gas mileage prior to modification. You then will make the modification to you car (fuel additive, engine contraption, etc) and the make the same trek, using the exact same method. Your new gas mileage will be compared to your old one. Since the only thing that changed was your modification, it can safely be assumed that your modification is the reason for gas mileage increases or decreases.

It is vital that you also maintain the same speed, have the same tire pressure, etc for both trips. Keep as much the same as possible.

Report your results on the thread dedicated to your experiment. When enough people have weighed in their results, statistical analysis will be run and the final report will be written up.

[bagpipe goatee]

< listen to this, matt definitely did his homework with coming up with this. however, with my civic at the gas station i go to, it clicks off at a different place every time, so the more trials the more accurate.

[Matt Timion]

This is one of those elements of error that we really cannot control. That is one of the reasons I want to have five trials per person. This is also why I want everyone to go to the same gas station. While it isn't totally fool proof, doing everything in our power the same way will in fact help reduce error.

If it's off by 1/2 gallon, that's just the way it is. That means it's also off by 1/2 gallon before we do the modification. In the end I think it will all balance out.

Then again one time I was filling up our 87 CRX Si and the stupid pump kept clicking after 3 or 4 seconds. It persisted no matter how much I tried to stop it. In a case like that I'd recommend you go to a different pump.

In all reality, the more trials the better. If you feel up to it, make 10 trips instead of 5. ;)

[bagpipe goatee]

sorry, i missed where you said "five trials." I'm beat and i'm also not feeling the best. don't you just love online complaining?

[beatr911]

Matt, thanks for getting this experimentation thing going. Not sure if your are busy (as we all are) or the idea is fading a little, but setting up a database where we all can post our trials and results to including the specific conditions of the experiments would be very useful to us all.

This is just a post to keep the idea alive and a vote for a database we can populate.

[SVOboy]

Haha, you're amazing, he's been spending a lot of time, and he's almost done it, I wouldn't be surprised if it's up in the next day or say, just stick around, ;-)

[Matt Timion]

i've been super busy lately with work. I'm currently working on a mpg database so people can keep track of their gas mileage on the website. I'm also planning on doing an experiment specific database for result posting of experiments.

As the website grows more and more people will use it and it will be a great source of information for future users.

[beatr911]

Thanks Matt, it's about time someone accumulated this kind of data so we all can benefit from each other. Kudos!

[Matt Timion]

Thanks Matt, it's about time someone accumulated this kind of data so we all can benefit from each other. Kudos!

I spent a few hours working on the MPG database yesterday. With any luck it will be done today. I will then tackle the experiment database.

[Ted Hart]

This is one of those elements of error that we really cannot control. That is one of the reasons I want to have five trials per person. This is also why I want everyone to go to the same gas station. While it isn't totally fool proof, doing everything in our power the same way will in fact help reduce error.


Matt- Kudos on your research procedure! Very well done! There are always improvements (temperature, humidity variations), but %-wise, they're not too large a source of error. Stressing averages is good, too!:thumbup:

[GeneW]

A few more points....

1. Keep careful records, especially of external factors like ambient temperature, precipitation, number of passengers (including estimates of their weight), type of gasoline (including vendor) and whether any maintenance was recently done on the car (especially performance related issues like tune ups, ignition system changes, or fluid changes).

2. As one of my former supervisors at a Fortune 25 R&D lab said to me, "We let the data speak to us". Record your data with appropriate notes and do not draw any conclusions until you've formally finished the tests.

Sometimes things aren't what they seem. Sometimes there are unknown effects which matter more than what you tried to change. Sometimes these unknowns can only be inferred from the data itself.

Don't fall in love with any ideas, especially your own ideas. Be ready to discard anything if it doesn't make sense.

Richard Feynman put it best - real science involves putting anything into your report that would tend to falsify your results. If you poured a bottle of dry gas into the tank record it. If you had bunions one day that made your feet light on the pedal record that day, if you saw the gasoline tanker truck pulling out of the gas station record it. If you got stuck behind a lumbering garbage truck which drafted you. Etc.

Gene

[WorkingOnWise]

I'm going to do some formal tests on several products and methods this summer, as part of a business plan, and I have some questions that I think the collective wisdom on this forum can help with.
I have a 75 mile stretch of highway picked out that I can travel uninterrupted on pretty much anytime I need to, and I will run down this 75 mile stretch and back, giving me a 150 mile test trip.
I will begin and end each test run by driving a 15 mile highway loop then filling up at the same pump at the same gas station.
My questions are these.

1) Is topping off the gas tank, up to the top of the filler neck, really a good way to be sure there is the same quantity in the tank for every test? My thinking is that the fuel is aerated as it flows out of the nozzle, and this foam can give a minor variation in actual fuel quantity. I have seen a variation of a gallon or more when the nozzle shuts it self off. What do you think?

2) Do I need to try to match the atmospheric pressure, ambient air temperature, and relative humidity levels of my baseline tests? Or is this too anal?

3) How close should the vehicle weight be to the baseline to ensure measurable accuracy. The van weighs 3800 lbs. Is a 200 pound variation enough to skew the results?

4) Is there any measurable benefit to tires with lower rolling resistance? Maybe narrower tires that the OEM installed, as long at the overall height remains correct? If so, where can I get information on tires that give the best MPG, or that have the lowest rolling resistance?

Thanks
Keith

[theholycow]

WorkingOnWise,

1. I think most people on this forum get the most consistent results when they stop pumping without topping off at all. I vaguely recall reading about someone who uses a small bottle of gas and weighs it before and after the experiment or something. How do you know the "gallon or more" variation you saw was pump error?

2. Yes. Air temperature is very important, and I suspect the others are too.

3. Why would the weight change during the experiment? It should be the same.

4. Narrower tires have higher rolling resistance at the same pressure (assuming they're the same model tire), because they have to deform the sidewall more to make the same size contact patch (which you will get when you have the same pressure and weight). They do have less aerodynamic drag, though. I've posted a few lengthy posts explaining this in a little more detail; I really need to put together a canned post for it...
http://www.gassavers.org/showpost.php?p=100399&postcount=7
http://www.gassavers.org/showpost.php?p=99043&postcount=151
http://www.gassavers.org/showpost.php?p=101846&postcount=8

[WorkingOnWise]

theholycow,
The pump error I found was by filling to the point that the nozzle tripped off, noting the quantity of fuel, the topping off the tank, right to the top of the filler neck. I did this on several pumps and several cars a few years back, and saw some pretty wide variations. During this time I had a 1978 Olds 98 that had the fuel neck behind the license plate. That particular car would take around 3 more gallons after the nozzle first kicked off. I blamed it on the fact that the filler neck was on such a narrow angle that the foam from filing didnt get enough time to dissipate before backing up into the fill nozzle.
I have also seen pumps that tried to pump fuel so fast that almost as soon as I hit the lever, the nozzle tripped off, and there was still a 1/4 tank to go on the gas gauge.

The weight could change because if I decide I want company on a test run one day, and not the other, I gain or lose up to 200 lbs, depending on who goes with me :) 150 miles, plus 15 miles to warm the tires and drivetrain, plus time to fill up 2 times makes for about 3 1/2 hours of time. Good time for bonding with my boy...or listening to my music that drives the rest of the fam crazy! lol
If the weight will skew the results too much, I can compensate with sand bags when I ride alone.
The goal for me is to test in a real-world but consistent environment what works, and how well.

Thanks for the info on tires. I'll be reading that next!
Keith

[flapdoodle]

theholycow,
The pump error I found was by filling to the point that the nozzle tripped off, noting the quantity of fuel, the topping off the tank, right to the top of the filler neck.

My filler is on the side, and I found that sometimes I can pump a little more fuel in if I bump the side of the car with my hip to make the fuel in the tank slosh. I have done this so many times it is just habit now. Sometimes it is enough that I can hear it sloshing.

The concrete around the pumps are sloped slightly for water runoff, so the side of the island I choose will determine it the filler neck is high or low relative to level.

[theholycow]

The pump error I found was by filling to the point that the nozzle tripped off, noting the quantity of fuel, the topping off the tank, right to the top of the filler neck. I did this on several pumps and several cars a few years back, and saw some pretty wide variations. [...]

But, did you try it and find it inconsistent on the same car, same pump, same day? That's the accepted way to call it "accurate" around here, and I'd be interested to hear if that assumption has been experimented and proven wrong. It wouldn't be the first time for me. :)

Thanks for the info on tires. I'll be reading that next!

I finally made the canned post, so I'll have it ready for next time. :thumbup:

[WorkingOnWise]

Ya know, I never did do a conscious test on the same car, same pump, same day to be sure. I figured if there is fuel at the top of the filler neck, it's pretty hard to say it isn't full yet...but not knowing where the air pockets might be on the top of my fuel tank, if the angle of the vehicle were right, the location of an air pocket can change too...so my method of full being full could easily be flawed too. Bummer!
Well, I have been playing with the idea of a installing a 6 gallon auxiliary fuel tank to use for fuel additive testing, so I only have to burn off 6 gallons of gas instead of 20 before I can test the next additive. Now with this discussion of full or not, I will definitely add the auxiliary tank, set up with a switch so I can flip between them as needed, like the F series ford trucks have had forever. Then, I can control the shape of the tank, and be sure that I start out with the exact amount of fuel each time, no matter what the pump might say or do. Now the trick is to find a good place to install in under a 1995 Villager!

[Roadkill]

All I will say if you are going to do a gas millage test , do it the old way of how much gas it took to fill up your fuel tank, then drive till it is empty note how much city and open highway driving one did. then fill up your fuel tank again, subtrackthe amout of gallens it took to fill it up again to get the total gallens of gas used. then divid the gallens by the miles traveled to get your real fuel millage. And please Stop using that stuped scan gage for it does not tell the real millage.

[Hateful]

I've yet to see any difference in mileage with the scangauge and my trip meter on the instrument panel; accept for a few miles where I Engine Off Coasted without having the gauge set to Hybrid. My gas logs have always been as described above. The tank MPG on the scangauge has been off before but it's still a good tool for testing.