@repligate 2. is kind of weird, it is stateless - if you send the request to a different instance with no shared kv cache, with the right settings you'd get the same response. there's not a stateful cache, it's just the nature of attention to have the embeddings for the tokens so far
in reply to: 1965671097048998078
same thread: 1965671097048998078 1965684324340314282 1965685095404343702 1965686065517527245 1965687300870078490 1965689520244101453 1965690945242103822 1965691665920000378 1965692428436038089 1965694514083102956 1965736347307807091
Reproduced against link rot, credited and linked to its original. Yours and you’d rather it weren’t here? Open an issue.