The dose of Potassium Cyanide that would be fatal in 50% of those ingesting it is ~5 mg/kg. For an average 70 kg person that's 350 mg. Not a lot but not an "OMG I just swallowed 1 mg of cyanide!".
...
I wonder if the Same Mindset could/should be used when people consider that they May be inhaling Aldehydes?