Hi everyone,
Anyone have any suggestions on why the red lane is turning blue? This image is the start line image from yesterday.
I haven't done anything on purpose to change the colour.
I don't know if it continued through the session as I have turned off image logging as it seemed to be causing the non-start issues I've been dealing with this season.
Images:
I've never seen red/blue swap like that but I do wonder quite often about a greyish band in the bottom of the images in the same region where in your image the red/green are correct. I wonder if they might be related.
Seems that there are issues with the latest sd card image according to the email we got earlier. From the email "Good news: fixed. Bad news: not able to rerun tests."
Thanks, Pico. Do you have an image of that grey bar you could share?
Not sure if the sd image problems would cause this. If it were, I would have expected others to have seen it as well. It looks like the colour correction is changing the reds instead of just boosting up the various red tones. Which is very odd since I've not changed any of the image correction code from the base code.
I tested it here and got the same result so it's not hardware related. I cloned the base code repo and ran that and got the same results again. I've not had time install an older version of raspbian yet, but that may fix the problem.
Hi Jamie,
I had the same issue during the last race. It caused my bot to behave strangely. The red and blue channels have been swapped at the 72nd pixel. It feels like a cv2 or driver error.
I save my images from inside ImageProcessor.py just after they have been acquired. I save one image every half a second. I initialize two variables in their respective files: Globals.nextTimeToSave = 0.0 and Settings.photoInterval = 0.5 . See picture below for code and location.
I believe I see the same thing Pico describes. It's even visible in the image set available on the blog. I've attached some pictures taken from my runs, and one that appears to be some testing that was performed when investigating the failed start from Test 2. It appears to have a dark region in the lower third of the image. I thought initially some code was performing color or brightness correction, but I can't verify that and it doesn't appear to affect images from the simulator.
I believe I see the same thing Pico describes. It's even in the image set available on the blog. I've attached some pictures taken from my run in race 1, and one that appears to be some testing that was performed when investigating the failed start from Test 2.
It appears there is a dark region in the lower third of the image. First thought is that code is performing color or brightness correction before the image is being saved, however I can't see any obvious sign of it in images taken from the simulator. And as stated above, it's in their official image set, so it indicates it happens even with the base code.
Hi yes, sorry me calling it a grey band might be a bit misleading, more like a grey distortion, like a brightening or darkening in the lower part of the image. I was also going to mention that the official image sets show it quite clearly. I've never been able to recreate the issue on a pi3b+ with a v1 camera, but that isn't the race spec of pi 3b and v2 camera, so not a true comparison. I have seen it in a lot of the images I've received back in logs.
It seems that the someone at the track end has decided that reverting to the previous software installation solves all the current problems, so let's go with that for now. To me this seems a bit extreme. I've always used the most up to date raspbian installation I can get (I was testing using stretch last season while the robots used jessie but err yeah also we came last and spent more time crashing than doing laps so err yeah), and if I recall the standard formula pi installation requires a 'sudo apt-get upgrade' as part of the process, so rolling back to the 2016 sd card image seems a bit extreme. The main difference seems to be using debian jessie rather than debian stretch, but I don't know enough to tell you why that would affect the python/numpy/opencv/thunderborg combination we all seem to have in common. V4L2 which opencv depends on for image capture might also be a suspect, again I don't know.
From a bit of reading around, it seems fluorescent lights might cause issues with digital cameras. The track does have a LOT of fluorescent lights.
This is my therory though, totally unproven, and without wanting to be told off for "negative comments". The raspberry pi 3b recommended power supply is 5V 3A, and the camera module adds another 0.25A. The monsterborg chassis uses the BattBorg power supply which is rated at 5V 1.5A. I have a hunch that the ThunderBorg controller was designed to fit a Pi zero, at least it fits the dimensions and mountings better than a Pi B. So the 1.5A power supply would suit a Pi zero, but it may cause problems with the Pi 3B.
I'm not sure I've come anywhere close to answering the original question but thanks for giving me the excuse to have a rant. I do hope that everyone who has been affected by this is allowed to rerun their tests. I would strongly encourage everyone to lobby for abandoning the current season and awarding prizes based on the current standings, it might be Pico HB's only hope. ;)
Jon
I don't think its hardware or florescent light related. Neither of those things have changed and they would have been affecting everyone instead of just some of us.
I hope to get some time this week to roll back the OS and test using my setup here.
As for abandoning the season, I won't be joining you with that request. Everyone at FormulaPi is giving 110% and although its unfortunate, these problems are no one's fault. They have been nothing short of amazing when it comes to helping those of us who don't start. Even staying after the stream and using more of their free time to help find the problem.
Everyone who has responded has helped answer my question. I wanted to know if others have seen similar results to mine, or if I had made a mistake in my code.
I will follow up later with the results of rolling back the os.
After some investigation it would appear that the Pi camera is having issues at some resolutions, but not others, with the new SD card load.
I have attached some images of a colour "test card" taken with both the old and new image at a few sizes to show what is happening. The smaller images have been software scaled to make them easier to see.
Our best guess is that the "grey line" is a related problem since the error is in the same place, it just does not have the same level of effect as the red / blue swap does. The error is always in the same place (even new vs old) for a given size, also some sizes have no problems at all. In fact if you look at the black and white lines on the new images you can see that the "grey line" problem is in fact present as well...
We do not see the same problem with
raspistilltaking photos at these sizes, so I do not think it is an issue with the camera itself. It is more likely to be a problem with either the Video4Linux driver or with OpenCV accessing the camera in some regard. To hazard a guess I think there might be a mix-up somewhere between BGR and RGB formats (which are both common).For reference these tests were done with just a monitor, camera, keyboard, and mouse attached (no robot). The Pi was powered from an official Raspberry Pi mains supply, so it is not a power issue.
I have also taken an image with all the lights off using the old SD card. This shows that while the camera can barely see anything it still has a distinctive line. I took another with the camera completely covered and still get the same issue. This eliminates any potential lighting issue, it must be some kind of software or driver problem...
Going back to the old image may seem drastic, but it definitely solves the red / blue swap (even though it has a problem) and it is what everyone has been running with for a while now. Ideally we would look into the problem now, but we do not even have the time to re-run tests this week :(
Anyone who wants to rebuild the old SD card image can find the instructions here: Old Formula Pi SD card image - Monster series. The instructions do not upgrade the Raspbian in that image, they only do an apt-get update so that it is able to download the missing packages.
BE WARNED: This image does not work with the 3B+, which is why we updated in the first place. We will try and get a 3B+ compatible image working properly before next season starts.
I copied Sheldon's Fury's experiment from above and swapped the red and blue channels only on the best quality image taken. The result (below) is essentially the same as the old SD card image for that size.
Thank you Arron for all your hard work, it is always good to hear it straight from the horse"s mouth.
Jamie, I'm sorry for the confusion, my last sentence about abandoning the season was meant to be a joke. As in that would be the only way PicoHB could hope for winning a prize.
Sheldon, awesome test rig. Genius well done.
I don't want people to think that I want any changes made to the hardware, I have a monsterborg chassis of my own and they are excellent. (There is some room for improvement around the electrics and layout but hey.) The robots used on the track should be exactly the same kit that anybody can buy for themselves from piborg (and the piborg gang could/should do more to promote their sales).
If the knowledge that I am carrying on developing with 3b+ and strech (standard image after a dist-upgrade) is of any use to anyone then I'm keen to help.
I ran an experiment earlier. All of the code is in the attached compressed file. My findings follow and more details about the experiment are at the bottom.
FINDINGS
It is cv2 that is causing the error because our frame height is not divisible by 16. Frame sizes not divisible by 16 result in cv2 using a resizing algorithm that goes bonkers when resampling decimals. Some frame sizes like 127 also appear normally and I assume this is because it gets rounded up to 128 during resampling. Please try to duplicate my results. The image below shows examples of my findings.
We have two choices:
1. change our frame height to a size that is divisible by 16, example 128 instead of 120, or
2. change the code so that we use the picamera and picamera array modules to pull frames from.
EXPERIMENT
One file tests cv2's frame sizes and the second file tests if the strange artifacts are visible using picamera and picamera array.
Run test cv2 capture.py using the original frame size settings to confirm that the undesirable artifacts exist. These undesirable artifacts can be the red and blue channels partially or full swapping or can be a darken portion of the screen.
Then comment out the original frame size variables and uncomment the frames divisible by 16 variables. You should see that the artifacts are gone. Feel free to test other frame sizes. Frame sizes that are just below frame sizes that are divisible by 16 will sometimes result in frames without undesirable artifacts, e.g. 160x127. When the frame’s width is not divisible by 16 the whole image is affected by the undesirable artifacts.
Run test picamera.py and notice that the same undesirable artifacts do not exist.
Run both files again and comment out
os.system('sudo modprobe bcm2835-v4l2’)and notice that the results remain the same for all of the above.P.S.: My name is James. Sheldon is my Tibetan Spaniel.
Unfortunately I am not in the office this week, but I will give this a try when I am back on Monday.
I would suspect it is more likely that the Video4Linix driver is the problem for two reasons:
I think there is possibly a bit of confusion regarding this line:
os.system('sudo modprobe bcm2835-v4l2’)What
modprobedoes is to load a driver if it can find it and if it is not already loaded. Once it is loaded it stays loaded until you restart the system or you remove the driver using amodprobe -rcommand. You can use the commandlsmodto check what modules are currently loaded.If you are happy to do some more testing I would suggest commenting out the
modprobelines, then restart the Pi and try both scripts. It might be worth runninglsmodas well just to confirm that nothing else is loading the V4L driver. My guess is that the picamera version will work fine but the cv2 capture version will fail to talk to the camera.Either way the "divisible by 16" theory is an interesting one, especially since a lot of software bugs tend to be related to powers of 2. I am also curious if different aspect ratios make a difference, however the actual camera sensor is 4:3 so using a 16:9 ratio for example would crop part of the image.
On a side note the very first version of the Formula Pi code when we were still developing the YetiBorgs was using the picamera module. We moved to OpenCV because the frames from the camera were loading faster, which makes a big difference on a Pi Zero :)
@Arron, Thanks for replying earlier. I’m on holiday myself.
@ALL, I looked further into the problem and followed Arron’s suggestions. I did indeed misunderstand what modprobe did. I checked and if you comment out
os.system('sudo modprobe bcm2835-v4l2’)after restarting cv2 errors out. Picamera automatically initializes the same modules as cv2. The modules are listed below: (as visible in terminal using the commandlsmod):Module
bcm2835_v4l2
v4l2_common
videobuf2_vmalloc
videobuf2_memops
videobuf2_v4l2
videobuf2_core
videodev
media
The picamera documentation states in several places that the rpi camera module will round horizontally to a multiple of 16 or 32 and round vertically to a multiple of 16.
Even the bcm2835-v4l2 documentation has a special note that when the video capture format is set to 1080p it should have a height of “1088 not 1080.” 1088 is a multiple of 16.
After reviewing the cv2’s v2l video capture files (cap_v4l.ccp and cap_libv4l.cpp) it is noticeable that they do not take any rounding to the 16th bit into account.
I also took a look at the cv2 v3.4 code related to v4l and the files don’t seem to have taken the multiple of 16 into consideration as well and I expect it produces the same undesirable artifacts. I will check next week when I get back from my trip.
As for the cropping, it appears that most image sizes are cropped. The actual image sensor size is just larger than 4:3 by just 4 pixels on the vertical. This difference makes it divisible by 16.
The multiples of 16 seem to be important. What do you think?
I think you have hit the nail on the head :) The Pi camera V4L driver expects the user to request multiples of 32 and 16 or otherwise handle the difference, but CV2 expects the driver to return the actual resolution it can format to instead.
From the experiments this has always been an issue (if only minor). My guess is that one of the updated components has changed the default image format used and in the process made the pre-existing error a lot more obvious!
I am not entirely sure which component is "non-standard" in this case, but the Video4Linux documentation suggests that their is supposed to be some kind of negotiation for details such as frame rate and resolution.
From some quick testing of oddball resolutions it does look like sticking with the multiples works fine. Currently I am thinking that the code for next season should automatically adjust the resolution choice to the closest matching resolution and have a default of 160x128 or step it up to 192x144.
I think 192x144 is probably better as it keeps the same aspect ratio as intended (4:3) and the difference in performance should be minor.
Add new comment