I am constantly seeing google trying to access these links directly which generally results in a 404 because the file no longer exists because the cache has been cleared. Shouldn't these pages have a nofollow and/or noindex appended to the links?
I have no idea if it is hurting SEO but it seems google is trying to access the links just as much as any other.
I don't think they're links. They are included JS and CSS code (style and script tags). If someone is looking at a history page, that page could be referencing old cache data. But that would be somewhat rare.
Not smart, I suggest you remove it. Google parses pages and by disallowing access to those critical resources, you are making your page non-functional for their algorithm, meaning that they will see a page with no styles or JS.