BRN Discussion Ongoing

[Frangipani]

More details about the AIS 2024 Event-Based Eye Tracking Challenge in April, in which the BrainChip “bigBrains” team consisting of Yan Ru (Rudy) Pei, Sasskia Brüers Freyssinet, Sébastien Crouzet (who has since left our company), Douglas McLelland and Olivier Coenen came in third, narrowly trailing two teams from the University of Science and Technology of China:

https://iris.polito.it/retrieve/bb1c9055-5245-4c2a-b729-070106bfd772/AIS_2024_Event_based_Eye_Tracking_Challenge_Final_Report.pdf

View attachment 70431
View attachment 70432
View attachment 70433
View attachment 70434

Remember the AIS 2024 Event-Based Eye Tracking Challenge in April, in which the BrainChip “bigBrains” team consisting of Yan Ru (Rudy) Pei, Sasskia Brüers Freyssinet, Sébastien Crouzet (who has since left our company), Douglas McLelland and Olivier Coenen came in third overall?

I just discovered this on GitHub: TENNs-Eye, “a lightweight spatio-temporal network for online eye tracking with event camera, belonging to the class of TENNs (Temporal Neural Networks) models by BrainChip.”

Brainchip

Brainchip has 13 repositories available. Follow their code on GitHub.

github.com

 

[Frangipani]

Sounds like a bit of a game changer.
They should have called it BIGGUS DICKUS instead of CENTAURUS

GitHub - Brainchip-Inc/Centaurus: Deep SSMs with optimal contractions

Deep SSMs with optimal contractions. Contribute to Brainchip-Inc/Centaurus development by creating an account on GitHub.

github.com

"Centaurus is a deep SSM that allows for flexible channel connectivity (much like CNNs), which is enabled by optimal tensor contractions. It is an extension of our aTENNuate network, now adapted for several audio tasks such as keyword spotting, denoising, and ASR. Unlike Mamba however, it does not use data gated SSM matrices, yet"


This is from Rudy Pei LinkedIn post.
"As demonstrated, the Centaurus network is SOTA across many real time audio tasks, including keyword sporting, denoising, and speech recognition. More important, it is super lightweight and edge compatible with many devices, including BrainChip's Akida hardware. We see a future where this type of network will become ubiquitous on the edge".

Yet, another feather ditty to the AKIDA repertoire and not a fat lady in sight.

Are Centaurus and Mamba fighting or dancing together?

 

[Frangipani]

Beyond Traditional Security: Neuromorphic Chips and the Future of Cybersecurity​

Bradley Susser
·
Follow
10 min read
·
1 hour ago
https://medium.com/m/signin?actionU...b5---------------------clap_footer-----------

A New Era of Cyber Warfare​

The rapid proliferation of cyber threats across the digital world exposes the vulnerabilities of traditional computing architectures, which often rely on outdated signature-based detection methods against increasingly sophisticated attacks. Polymorphic malware, which constantly mutates its code, easily evades conventional signature-based detection, sometimes encrypting files for ransom. Furthermore, distributed denial-of-service (DDoS) attacks overwhelm networks, crippling performance and causing widespread outages. Insider threats, often difficult to detect with traditional security, require analysis of user behavior. In this article, we will explore the intersection of neuromorphic computing and cybersecurity, examining how these two fields can enhance each other and reshape our approach to digital defense.
Moreover, zero-day exploits (vulnerabilities unknown to vendors, giving them zero days to patch them) pose an especially challenging threat to traditional security systems. Neuromorphic systems, by learning from evolving patterns, enable faster, real-time identification of these vulnerabilities. This offers a significant advantage especially in defending against zero-day attacks.
Neuromorphic computing, inspired by the structure and functionality of the human brain, offers a new approach to cybersecurity, providing enhanced threat detection, adaptive defense mechanisms, and energy-efficient real-time processing. These systems learn malware behavior, enabling them to identify threats even with altered signatures, and analyze network traffic patterns in real-time to identify and mitigate DDoS attacksmore effectively than traditional methods. Learning baseline user behaviorallows these systems to identify anomalies indicative of malicious activity. Neuromorphic computing can also enhance cryptographic algorithms for improved data protection and operational efficiency.

Understanding Neuromorphic Computing​

In cybersecurity’s rapidly evolving landscape, speed is key. Neuromorphic computing excels in this area, mimicking biological neural networks to process information in parallel and learn from experience. Think of how your brain instantly reacts when you see a car suddenly brake in front of you, hear a loud noise behind you, or smell smoke. Your eyes, ears, and nose are constantly feeding information to your brain, which processes it all simultaneously and allows you to react in real-time with very little effort. Neuromorphic computing works similarly, using artificial neurons and synapses on specialized chips to make very fast decisions. These chips perform complex computations far more efficiently than conventional computers, which process information one step at a time. Consider, for example, how traditional systems using CPUs and GPUs often rely on sending data back and forth between edge devices (like smartphones, security cameras, or smart home appliances) and the cloud for processing. This constant communication requires significant energy. Even within a single device, CPUs and GPUs must repeatedly access separate memory locations, a process that becomes increasingly energy-intensive with the enormous data models used by AI and large language models. These massive models, growing at such a rapid pace, demand tremendous amounts of energy for processing. In contrast, neuromorphic computing, with its parallel processing and on-chip memory, offers a much more energy-efficient approach. This speed and efficiency make neuromorphic computing particularly valuable for enhancing cryptographic systems and ensuring data protection (e.g., quickly identifying and blocking malicious network traffic).

Real-World Applications and Pilot Projects​

While neuromorphic computing is still in its early stages of adoption in the cybersecurity field, several promising pilot projects and initiatives are already demonstrating the power of chips like Loihi, NorthPole, and Akidain real-world scenarios. Take for example BrainChip’s Akida. One of the most notable applications of Akida is in the field of edge security. Early partnerships with IoT device manufacturers are testing Akida’s ability to handle real-time threat detection in resource-constrained environments. These pilot projects focus on Akida’s ability to process large volumes of data at the edge, detect anomalies, and mitigate threats without requiring a constant connection to a centralized server. These projects are paving the way for more efficient, scalable, and secure IoT networks in industries like smart homes, industrial IoT, robotics and autonomous vehicles.

In the case of IBM, the company has integrated its NorthPole chip into several smart city initiatives to combat the growing challenge of securing urban infrastructures against cyberattacks. NorthPole’s energy efficiencyand low latency make it ideal for monitoring critical infrastructure such as traffic systems, power grids, and public surveillance networks in real-time. Pilot projects in cities like New York and Berlin are exploring the chip’s ability to detect cyber threats as they happen, enabling more proactive and automated responses to attacks.

Finally, Intel’s Loihi 2, which is being trialed in the financial services sector, is helping banks and financial institutions strengthen their cybersecurity defenses. Loihi 2 is currently being tested for fraud detection and real-time transaction monitoring by utilizing event-based, asynchronous processing. Asynchronous processing means that the chip doesn’t wait for all the information to arrive before starting to work; it processes data as soon as it becomes available, like a team working on different parts of a project at the same time instead of waiting for each person to finish their task before moving on. In a pilot program with a major financial institution, Loihi 2 has shown promise in quickly identifying fraudulent transactions by learning transaction patterns and flagging irregularities, even as financial data changes dynamically.

These early applications demonstrate how neuromorphic computing is not only a theoretical concept but a practical, scalable solution for improving cybersecurity systems. As these technologies mature and prove their value, we can expect to see further integration into industries that rely heavily on data security, such as finance, healthcare, and critical infrastructure.

Neuromorphic Computing and Cryptography​

Neuromorphic computing can significantly enhance both the performance and security of cryptographic systems, playing a critical role in protecting data at rest and in transit. Its ability to process information simultaneously (parallel processing) and to react instantly to specific triggers (event-driven computation) makes it ideal for speeding up cryptographic tasks and creating advanced cryptographic methods.

Neuromorphic computing, with its ability to process information in paralleland react to events as they happen, has huge potential for improving the security of encryption. When data is stored (“data at rest”), these systems can automatically update and strengthen encryption, making it much tougher for hackers to guess the codes or use brute-force attacks (trying every possible combination). Think of a brute-force attack like trying every possible sequence on a lock. Hackers use powerful computers to try and guess passwords or encryption keys. Neuromorphic systems make this much more difficult by constantly changing the encryption key, so it’s like a moving target for these adversaries. Because they can adapt, these systems can also respond instantly to new threats, even those from quantum computers.

Regarding “data in transit,” a key challenge is ensuring secure communication without introducing latency, which refers to the delaybetween sending and receiving information. Neuromorphic systems can accelerate encryption and decryption processes, minimizing this delay and ensuring secure, real-time data transfer. This speed is crucial for applications where timely communication is essential, such as video conferencing or online transactions. Furthermore, by handling cryptographic tasks independently and concurrently, these systems can efficiently manage high-volume data streams without straining network resources, further reducing potential bottlenecks (congestion) that could introduce latency..

Looking ahead, the rise of quantum computers poses a serious threat to current online security. These powerful machines could crack the codes in seconds as opposed to utilizing classical computers where it would take years. This is where post-quantum cryptography (PQC) comes in. PQC involves new, super-complex math problems that even quantum computersshould struggle to solve. However, these new methods are also very resource intensive, meaning they take a long time to run. Neuromorphic chips, with their speed and efficiency, can accelerate these PQC calculations, making them practical for everyday use and ensuring our datastays safe in the quantum era.

Another exciting area is homomorphic encryption. Imagine being able to perform calculations on encrypted data without ever having to decrypt it. That’s homomorphic encryption. It’s like having a secure box where you can manipulate the contents without ever opening it. This is great for privacy, but it’s also incredibly complex and slow. Neuromorphic computing, because it’s good at handling complex calculations, could make homomorphic encryption much faster and more practical, opening up new possibilities for secure and private computing.

Challenges and Hurdles in Neuromorphic Computing for Cybersecurity​

While the potential of neuromorphic computing in cybersecurity is clear, several key hurdles must be addressed before widespread adoption. One primary challenge is the lack of mature software frameworks. Unlike traditional computing, neuromorphic systems require specialized programming models not widely available, necessitating the development of new tools, compilers (which translate high-level programming languages into simpler machine code that computers can understand), and debugging environments (which help programmers find and fix errors in their code). This requires significant time and expertise, slowing integration. Furthermore, specialized expertise is crucial for designing, programming, and maintaining these systems. Neuromorphic computing blends hardware, neuroscience, and AI, demanding multidisciplinary understanding. This specialized knowledge limits the pool of qualified professionals, thus impeding growth and limiting widespread adoption.
Neuromorphic systems are also vulnerable to adversarial attacks. Their adaptive nature could be exploited, with nefarious individuals developing attacks to manipulate the learning process. Several specific attack types pose concerns. Backdoor attacks incorporate subtle triggers into the system’s learning, causing misclassification or missed threats. Evasion attacks use artificially generated data to confuse the system’s decision-making, causing it to miss malicious activity. Model poisoning manipulates training data to impair learning, potentially allowing attacks to go undetected. Catastrophic forgetting attacks cause the system to forget critical patterns, reducing effectiveness. Physical layer attacks exploit neuromorphic hardware vulnerabilities using side-channel attacks (attacks that exploit unintended information leaks from a system). Defending against these sophisticated attacks requires specialized countermeasures like adversarial training and Improved system reliability and resilience.

Cost is another significant factor. While offering impressive energy efficiency and speed, neuromorphic chips like NorthPole, Loihi 2, and Akida are currently expensive to develop and deploy. High research and development (R&D) costs and specialized manufacturing processes hinder large-scale deployments. Embedding these chips into existing infrastructure could also require substantial investment and CAPEX (capital expenditures). Although exact pricing is often undisclosed, neuromorphic chips are generally more expensive than traditional processors due to specialized design, limited production, and highly advanced manufacturing techniques. For example, widely used GPUs like the Nvidia H100 have more predictable pricing compared to neuromorphic chips, which are still being refined.

Despite these cost challenges, neuromorphic computing is poised for greater affordability. As manufacturing improves and demand increases, costs are expected to decline. Economies of scale, coupled with growing industry adoption, will drive down prices in the coming years, particularly with the development of standardized software frameworks.

Future Research Directions in Neuromorphic Computing for Cybersecurity​

As neuromorphic computing continues to evolve, several exciting areas of research could vastly improve upon its application in cybersecurity. One of the most pressing challenges is designing powerful defenses against attacks. Researchers are exploring new methods for adversarial training, a process where systems are exposed to adversarial examples during their learning phase, to build more resilient models. Additionally, new ways to defendneuromorphic systems are being researched, like automatically updatingtheir defenses and using special programs to spot unusual activity. These could help the systems recognize and respond to new threats more effectively. These defenses, much like established cybersecurity frameworks such as those from ISO, NIST, ISACA (and others), aim to provide a structured and comprehensive approach by mitigating risks. Implementation of such defenses, similar to the adoption of robust frameworks, is crucial in ensuring system security.

Another critical area of research is the development of standardized software frameworks for neuromorphic computing. Currently, there is a lack of mature, user-friendly tools for programming and debugging these systems. Establishing standardized platforms, much like the development of standardized cybersecurity frameworks (e.g., the NIST Cybersecurity Framework, ISO 27001, or frameworks addressing cybersecurity within broader IT governance frameworks such as ISACA’s COBIT), would make it easier for developers to design and implement neuromorphic solutions across various cybersecurity applications. These frameworks could streamline the development process, enabling faster adoption and integration into real-world systems, similar to how established frameworks facilitate consistent and effective security practices.
Beyond current use cases in threat detection and cryptography, future research could unlock new applications for neuromorphic computing in cybersecurity. For instance, neuromorphic systems could be applied to biometric authentication, improving the speed and accuracy of fingerprint, facial recognition, and voice identification systems. Researchers are also investigating how neuromorphic systems could be leveraged in autonomous defense mechanisms for IoT devices, providing real-time, localizeddecision-making and threat mitigation at the edge. Just as organizations rely on frameworks like those from ISO, NIST, and ISACA (including their COBIT framework for IT governance) to guide their security practices, the development of standardized tools for neuromorphic computing will be essential for its effective deployment.

As the landscape of cybersecurity changes with the rise of quantum computing, future research could explore hybrid systems that combine neuromorphic computing and quantum computing to create robust, scalable, and quantum-resistant cryptographic protocols. This fusion of technologies would not only advance the field of cybersecurity but also ensure that systems can stand the test of future technological advancements. An integrated approach to security, much like the layered security promoted by various frameworks, will be critical for protecting against evolving threats. Exploring these research directions will allow the cybersecurity arena to benefit from more powerful, adaptive, and energy-efficient security systems capable of countering future cybersecurity threats, much like how adherence to established cybersecurity frameworks strengthens an organization’s overall security position.

​

Conclusion​

Neuromorphic computing holds immense promise for cybersecurity, providing real-time, adaptive, and energy-efficient defenses against increasingly sophisticated attacks. Systems like NorthPole, Loihi 2, and Akida illustrate this potential. Through massive parallelism, low-latency processing, and the ability to incorporate memory and compute on-chip(placing processing capabilities directly on the chip with memory), these systems enable faster, more effective detection of anomalies, DDoS attacks, and malware, ensuring that cybersecurity frameworks are ready to combat tomorrow’s challenges.
Neuromorphic computing, with its impressive speed and energy efficiency, is poised to secure digital infrastructure and advance future technologies. As these technologies mature, they will undoubtedly play a pivotal role in shaping the cybersecurity domain of tomorrow.

References​

Zahm, W., Nishibuchi, G., Jose, A., Chelian, S., & Vasan, S. (2022). Cyber-Neuro RT: Real-time Neuromorphic Cybersecurity. Low-Power Cybersecurity Attack Detection Using Deep Learning on Neuromorphic Technologies, CSIAC 2024: Volume 8 Issue 2.

ER, K. (2024). A Neuroscience Perspective on AI and Cybersecurity. ISACA Journal, 2025 Volume 1.

Cao, B., Zhang, Z., & Liu, W. (2017). A Survey of Neuromorphic Computing and Neural Networks in Hardware. Journal of Neural Networks, 29(3), 213–231.
Büchel, J., Lenz, G., Hu, Y., Sheik, S., & Sorbaro, M. (2021). Adversarial attacks on spiking convolutional neural networks for event-based vision. IEEE Transactions on Neural Networks and Learning Systems, 32(4), 1234–1249.
Modha, D. S., Akopyan, F., Andreopoulos, A., Appuswamy, R., Arthur, J. V., Cassidy, A. S., Datta, P., DeBole, M. V., Esser, S. K., & Ueda, T. (2023). Neural inference at the frontier of energy, space, and time. Science, 382(6668), 329–335. DOI: 10.1126/science.adh1174
Cybersecurity

Technology
Neuromorphic Computing
Future

https://medium.com/m/signin?actionU...b5---------------------clap_footer-----------

Written by Bradley Susser​

38 Followers
·570 Following
Masters in Info Systems. Real Estate Agent. Previously founded a consulting firm specializing in IPOs, capital raising, investor relations for over a decade
Follow

 

[manny100]

I think different markets will start to develop within the 'Edge' industry. Some chips will be more suited to some tasks than others.
It appears that OSS and AKIDA have different strengths and will be suited to different use cases.

 

[Frangipani]

D&R IP - SOC Days April 24, 2025 Santa Clara, CA

D&R IP - SOC Days April 24, 2025 Santa Clara, CA BrainChip will present and exhibit at D&R IP-SoC Silicon Valley 2025 on April 24 in Santa Clara, CA, a premier event dedicated to Silicon IP and IP-based electronic systems. Showcasing Akida technology, BrainChip will demonstrate how its...

brainchip.com

 

[TheDrooben]

One of our Integration Partners Teksun launching new services.......

Introducing Teksun’s ML-powered predictive Services– delivering next-gen intelligence with pattern recognition, AI, and predictive analytics for scalable and data-driven applications. | Teksun Inc

Introducing Teksun’s ML-powered predictive Services– delivering next-gen intelligence with pattern recognition, AI, and predictive analytics for scalable and data-driven applications. Our solutions optimize anomaly detection, algorithm design, and future forecasting to enhance efficiency and...

www.linkedin.com

Happy as Larry

 

[Smoothsailing]

One of our Integration Partners Teksun launching new services.......

Introducing Teksun’s ML-powered predictive Services– delivering next-gen intelligence with pattern recognition, AI, and predictive analytics for scalable and data-driven applications. | Teksun Inc

Introducing Teksun’s ML-powered predictive Services– delivering next-gen intelligence with pattern recognition, AI, and predictive analytics for scalable and data-driven applications. Our solutions optimize anomaly detection, algorithm design, and future forecasting to enhance efficiency and...

www.linkedin.com

View attachment 77074
View attachment 77077

Happy as Larry

Come on Brainchip

 

[RobjHunt]

Hi RobjHunt,

you unknowingly addressed a problem here that keeps surfacing time and again when translating the German word sogenannt into English, as the literal translation so-called has a negative connotation in English, indicating that the speaker/writer thinks a word used to describe something or someone is actually wrong.

However, this is not necessarily the case in German. And certainly not in this context. While the adjective sogenannt can be used sarcastically or as a way to distance yourself from something others say you don’t agree with, it does not convey any pejorative connotation in general, especially not in a scientific-technical context such as this MB LinkedIn post. It is commonly used in texts aimed at a broad audience to refer to specialist nomenclature nouns with a low lexical frequency in the general language, signifying to the readers in a perfectly neutral way that the following term they may not have come across before is the appropriate science-/tech jargon for it.

So in the context of the recent MB post, the word sogenannt in das sogenannte “Event-Based-Processing” simply means “this is how the (or we) experts refer to it”, and so-called - the literal translation into English - does not imply that the author thinks something is fishy about the term “event-based processing”.

Ok then

 

[7für7]

I hope for a nice price-sensitive announcement this week because it’s just the right time to take advantage of all the current attention on AI, low cost, and low power. If not, it might be too late again—maybe it already is, I don’t know. It’s kind of a pity because BrainChip has so much potential. Wishing everyone a great start to the week! Fingers crossed that something gets released!

 

[Pom down under]

I hope for a nice price-sensitive announcement this week because it’s just the right time to take advantage of all the current attention on AI, low cost, and low power. If not, it might be too late again—maybe it already is, I don’t know. It’s kind of a pity because BrainChip has so much potential. Wishing everyone a great start to the week! Fingers crossed that something gets released!

View attachment 77078

 

[TECH]

Intel is on a really slippery slide........more bad news, maybe Mike Davies and the Neuromorphic Team are financed
by a different executive team, because from where I stand, Intel has all the signs of being a takeover target from
another entity who can keep up with the times, Pat got the arse, the new CEO is under pressure already !

Here's just one recent article, of numerous negative articles.......being a partner of Intel Foundries sounds impressive, but
check out their 13 Billion Dollar loss ?

Intel’s AI dreams slip further out of reach as it cancels its big data-center GPU hope, Falcon Shores

Intel also admitted that its Gaudi 3 AI accelerator has not achieved “meaningful” adoption.

finance.yahoo.com

 

[Smoothsailing]

Looks very impressive.
Kinda like what I hope for with BrainChip.
Couldn't find much in the way of specifications though and from the pictures looks like latest iteration of current tech.
Nothing wrong with having shares in up coming current solutions along with what maybe will replace a lot of it.
Still need to live whilst waiting for the commercial uptake of the next generation.
Hopefully we have backed the right horse here with neuromorphic design.
It's taking longer to get traction than I assumed it would.
I hope we haven't missed the boat and have got enough of our toe in, at the doors that matter.

In reference to Sean in this latest podcast he was fairly confident in the eyes, ears and Wrist applications. I think once we get a deal signed which Iam betting will % happen before the end of April we will all become less stressed over BRN.

 

[IloveLamp]

Bradley Susser on LinkedIn: #neuromorphiccomputing #cybersecurity #aihardware #braininspiredai…

Beyond Traditional Security: Neuromorphic Chips and the Future of Cybersecurity The rapid proliferation of cyber threats across the digital landscape exposes…

www.linkedin.com

 

[7für7]

Intel is on a really slippery slide........more bad news, maybe Mike Davies and the Neuromorphic Team are financed
by a different executive team, because from where I stand, Intel has all the signs of being a takeover target from
another entity who can keep up with the times, Pat got the arse, the new CEO is under pressure already !

Here's just one recent article, of numerous negative articles.......being a partner of Intel Foundries sounds impressive, but
check out their 13 Billion Dollar loss ?

Intel’s AI dreams slip further out of reach as it cancels its big data-center GPU hope, Falcon Shores

Intel also admitted that its Gaudi 3 AI accelerator has not achieved “meaningful” adoption.

finance.yahoo.com

Well… that’s fake neeeews

 

[Diogenese]

Some details of Qualcomm Snapdragon 8.3 with Hexagon:

https://www.notebookcheck.net/Qualc...-Processor-Benchmarks-and-Specs.762133.0.html

The cryo-CPU is based on ARM's v9.2 architecture and consists of three clusters: The first includes a fast prime core (ARM Cortex-X4) with up to 3.3 GHz. The second consists of three power cores with up to 3.2 GHz and two additional cores clocking at up to 3.0 GHz (all Cortex-A720). The efficiency cluster consists of two power-saving cores (Cortex-A520) that operate at up to 2.3 GHz. The SoC uses the integrated Adreno 750 for graphics acceleration.

View attachment 77051

Compared to the Snapdragon 8 Gen. 2, Qualcomm promises a 25 percent higher GPU clock rate, whose power consumption is said to have been reduced by 25 percent at the same time. Similarly, the CPU's performance is announced to be now a whopping 30 per cent faster and requires 20 per cent less power.

A special focus is also placed on machine learning (AI). The integrated Hexagon NPU is 98% faster than its predecessor and promises a 40% improvement in efficiency. Qualcomm's AI engine is the first to support multimodal generative AI models, including LLM, LVM and ASR, and can process up to 10 trillion parameters directly on the SoC. Image manipulation is said to be possible within fractions of a second. In addition, the NPU of the Snapdragon 8 Gen 3 is also capable of INT4 and Meta Llama 2.

Qualcomm have a big portfolio of AI patents so it's difficult to tell which ones are smokescreens. This is a fairly recent one relating to ML. It uses capacitive voltage accumulation (analog) blended with digital processing.

US2022414443A1 COMPUTE IN MEMORY-BASED MACHINE LEARNING ACCELERATOR ARCHITECTURE 20210625


View attachment 77052


If this is what Qualcomm are using, clearly they have found some work-arounds to negate the manufacturing variability problem of analog. However, such work-arounds usually diminish the theoretical advantages of analog.

Snapdragon has a fistfull of ARM processors, so it should be possible to run TENNs on Snapdragon. Wouldn't that be a dainty flower arrangement to place before the customer?

Some more on Qualcomm's Hexagon AI which is pretty promiscuous, sharing the AI workload selectively between GPU, CPU, and NPU:

https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Unlocking-on-device-generative-AI-with-an-NPU-and-heterogeneous-computing.pdf

6 Heterogeneous computing:

Leveraging all the processors for generative AI Generative AI models suitable for on-device execution are becoming more complex and trending toward larger sizes, from one billion to 10 billion to 70 billion parameters. They are increasingly multi-modal, meaning that they can take in multiple inputs — such as text, speech, or images — and produce several outputs.

Further, many use cases concurrently run multiple models. For example, a personal assistant application uses voice for input and output. This requires running an automatic speech recognition (ASR) model for voice to text, an LLM for text to text, and a text-to-speech (TTS) model for a voice output. The complexity, concurrency, and diversity of generative AI workloads require harnessing the capabilities of all the processors in an SoC. An optimal solution entails:

1. Scaling generative AI processing across cores of a processor and across processors

2. Mapping generative AI models and use cases to one or more cores and processors

Choosing the right processor depends on many factors, including use case, device type, device tier, development time, key performance indicators (KPIs), and developer expertise. Many tradeoffs drive decisions, and the target KPI could be power, performance, latency, or accessibility for different use cases. For example, an original equipment manufacturer (OEM) making an app for multiple devices across categories and tiers will need to choose the best processor to run an AI model based on SoC specs, end-product capabilities, ease of development, cost, and graceful degradation of the app across device tiers.

As previously mentioned, most generative AI use cases can be categorized into on-demand, sustained, or pervasive. For on-demand applications, latency is the KPI since users do not want to wait. When these applications use small models, the CPU is usually the right choice. When models get bigger (e.g., billions of parameters), the GPU and NPU tend to be more appropriate. For sustained and pervasive use cases, in which battery life is vital and power efficiency is the critical factor, the NPU is the best option.

Another key distinction is identifying whether the AI model is memory bound — performance is limited by memory bandwidth — or compute bound — performance is limited by the speed of the processor. Today’s LLMs are memory bound for the text generation, so focusing on memory efficiency on the CPU, GPU, or NPU is appropriate. For LVMs, which could be compute or memory bound, the GPU or NPU could be used, but the NPU provides the best performance per watt.

A personal assistant that offers a natural voice user interface (UI) to improve productivity and enhance user experiences is expected to be a popular generative AI application. The speech recognition, LLM, and speech models must all run with some concurrency, so it is desirable to split the models between the NPU, GPU, CPU, and the sensor processor. For PCs, agents are expected to run pervasively (always-on), so as much of it as possible should run on the NPU for performance and power efficiency.

As we know, Akida can multi-task, running different models on the one SoC.

Running AI on CPU or GPU necessarily entails the use of software.

 

[JDelekto]

Some more on Qualcomm's Hexagon AI which is pretty promiscupos, sharing the AI workload selectively between GPU, CPU, and NPU:

https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Unlocking-on-device-generative-AI-with-an-NPU-and-heterogeneous-computing.pdf

6 Heterogeneous computing:

Leveraging all the processors for generative AI Generative AI models suitable for on-device execution are becoming more complex and trending toward larger sizes, from one billion to 10 billion to 70 billion parameters. They are increasingly multi-modal, meaning that they can take in multiple inputs — such as text, speech, or images — and produce several outputs.

Further, many use cases concurrently run multiple models. For example, a personal assistant application uses voice for input and output. This requires running an automatic speech recognition (ASR) model for voice to text, an LLM for text to text, and a text-to-speech (TTS) model for a voice output. The complexity, concurrency, and diversity of generative AI workloads require harnessing the capabilities of all the processors in an SoC. An optimal solution entails:

1. Scaling generative AI processing across cores of a processor and across processors

2. Mapping generative AI models and use cases to one or more cores and processors

Choosing the right processor depends on many factors, including use case, device type, device tier, development time, key performance indicators (KPIs), and developer expertise. Many tradeoffs drive decisions, and the target KPI could be power, performance, latency, or accessibility for different use cases. For example, an original equipment manufacturer (OEM) making an app for multiple devices across categories and tiers will need to choose the best processor to run an AI model based on SoC specs, end-product capabilities, ease of development, cost, and graceful degradation of the app across device tiers.

As previously mentioned, most generative AI use cases can be categorized into on-demand, sustained, or pervasive. For on-demand applications, latency is the KPI since users do not want to wait. When these applications use small models, the CPU is usually the right choice. When models get bigger (e.g., billions of parameters), the GPU and NPU tend to be more appropriate. For sustained and pervasive use cases, in which battery life is vital and power efficiency is the critical factor, the NPU is the best option.

Another key distinction is identifying whether the AI model is memory bound — performance is limited by memory bandwidth — or compute bound — performance is limited by the speed of the processor. Today’s LLMs are memory bound for the text generation, so focusing on memory efficiency on the CPU, GPU, or NPU is appropriate. For LVMs, which could be compute or memory bound, the GPU or NPU could be used, but the NPU provides the best performance per watt.

A personal assistant that offers a natural voice user interface (UI) to improve productivity and enhance user experiences is expected to be a popular generative AI application. The speech recognition, LLM, and speech models must all run with some concurrency, so it is desirable to split the models between the NPU, GPU, CPU, and the sensor processor. For PCs, agents are expected to run pervasively (always-on), so as much of it as possible should run on the NPU for performance and power efficiency.

As we know, Akida can multi-task, running different models on the one SoC.

As I understand it, the NPU (which is easily confused for neuromorphic because of the 'N') is another dedicated processor that parallelizes computational operations with the CPU and GPU at lower power requirements.

Qualcomm's Hexagon NPU is designed to offload those computations from the other two and is optimized for vector, matrix, and tensor processing (basically a lot of matrix math). I ran across an interesting thread on Hacker News, where someone benchmarked the NPU and found that it was not quite as good as the CPU itself. Again, the NPU intends to achieve performance through parallelization at lower power. They have the code for their benchmarks here on GitHub.

I believe Akida could still be a strong competitor to Qualcomm's AI offerings, or even potentially a replacement for Hexagon for better real-time processing on power-constrained devices.

 

[TECH]

Intel lost almost $19bn last year

Intel’s revenue was down 7pc to $14.3bn (€13.78bn) in its fourth quarter, the company has said in its first earnings report since Pat Gelsinger was ousted as CEO in December.

www.independent.ie

Intel cancels AI chip, talks painful past and simplified future

Nearly double the foundry losses of 2023.

www.theverge.com

Like I say, I personally don't see Intel derailing our company anytime soon, do you Donald Duck ?

We're targeting the Edge, Intel is targeting the Edge of a Cliff...

 

[Diogenese]

Some more on Qualcomm's Hexagon AI which is pretty promiscuous, sharing the AI workload selectively between GPU, CPU, and NPU:

https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/Unlocking-on-device-generative-AI-with-an-NPU-and-heterogeneous-computing.pdf

6 Heterogeneous computing:

Leveraging all the processors for generative AI Generative AI models suitable for on-device execution are becoming more complex and trending toward larger sizes, from one billion to 10 billion to 70 billion parameters. They are increasingly multi-modal, meaning that they can take in multiple inputs — such as text, speech, or images — and produce several outputs.

Further, many use cases concurrently run multiple models. For example, a personal assistant application uses voice for input and output. This requires running an automatic speech recognition (ASR) model for voice to text, an LLM for text to text, and a text-to-speech (TTS) model for a voice output. The complexity, concurrency, and diversity of generative AI workloads require harnessing the capabilities of all the processors in an SoC. An optimal solution entails:

1. Scaling generative AI processing across cores of a processor and across processors

2. Mapping generative AI models and use cases to one or more cores and processors

Choosing the right processor depends on many factors, including use case, device type, device tier, development time, key performance indicators (KPIs), and developer expertise. Many tradeoffs drive decisions, and the target KPI could be power, performance, latency, or accessibility for different use cases. For example, an original equipment manufacturer (OEM) making an app for multiple devices across categories and tiers will need to choose the best processor to run an AI model based on SoC specs, end-product capabilities, ease of development, cost, and graceful degradation of the app across device tiers.

As previously mentioned, most generative AI use cases can be categorized into on-demand, sustained, or pervasive. For on-demand applications, latency is the KPI since users do not want to wait. When these applications use small models, the CPU is usually the right choice. When models get bigger (e.g., billions of parameters), the GPU and NPU tend to be more appropriate. For sustained and pervasive use cases, in which battery life is vital and power efficiency is the critical factor, the NPU is the best option.

Another key distinction is identifying whether the AI model is memory bound — performance is limited by memory bandwidth — or compute bound — performance is limited by the speed of the processor. Today’s LLMs are memory bound for the text generation, so focusing on memory efficiency on the CPU, GPU, or NPU is appropriate. For LVMs, which could be compute or memory bound, the GPU or NPU could be used, but the NPU provides the best performance per watt.

A personal assistant that offers a natural voice user interface (UI) to improve productivity and enhance user experiences is expected to be a popular generative AI application. The speech recognition, LLM, and speech models must all run with some concurrency, so it is desirable to split the models between the NPU, GPU, CPU, and the sensor processor. For PCs, agents are expected to run pervasively (always-on), so as much of it as possible should run on the NPU for performance and power efficiency.

As we know, Akida can multi-task, running different models on the one SoC.

Running AI on CPU or GPU necessarily entails the use of software.

When you find yourself in a hole ... keep digging:

So Qualcomm's Hexagon NPU evolved from a DSP:

Building our NPU from a DSP architecture was the right choice for improved programmability and the ability to tightly control scalar, vector, and tensor operations that are inherent to AI processing. Our design approach of optimized scalar, vector, and tensor acceleration combined with large local shared memory, dedicated power delivery systems, and other hardware acceleration differentiates our solution. Our NPU mimics the neural network layers and operations of the most popular models, such as convolutions, fully-connected layers, transformers, and popular activation functions, to deliver sustained high performance at low power.

So naturally there's a side tunnel to DSPs:

Digital signal processor - Wikipedia

en.wikipedia.org

Such performance improvements have led to the introduction of digital signal processing in commercial communications satellites where hundreds or even thousands of analog filters, switches, frequency converters and so on are required to receive and process the uplinked signals and ready them for downlinking, and can be replaced with specialised DSPs with significant benefits to the satellites' weight, power consumption, complexity/cost of construction, reliability and flexibility of operation. For example, the SES-12 and SES-14 satellites from operator SES launched in 2018, were both built by Airbus Defence and Space with 25% of capacity using DSP.[6]


I wonder if Airbus intends to swap its DSPs for SNNs?

 

[Diogenese]

As I understand it, the NPU (which is easily confused for neuromorphic because of the 'N') is another dedicated processor that parallelizes computational operations with the CPU and GPU at lower power requirements.

Qualcomm's Hexagon NPU is designed to offload those computations from the other two and is optimized for vector, matrix, and tensor processing (basically a lot of matrix math). I ran across an interesting thread on Hacker News, where someone benchmarked the NPU and found that it was not quite as good as the CPU itself. Again, the NPU intends to achieve performance through parallelization at lower power. They have the code for their benchmarks here on GitHub.

I believe Akida could still be a strong competitor to Qualcomm's AI offerings, or even potentially a replacement for Hexagon for better real-time processing on power-constrained devices.

Yes. The Qualcomm white paper above states that their NPU evolved from the DSP, and it looks like they are sticking with it.

"Building our NPU from a DSP architecture was the right choice for improved programmability and the ability to tightly control scalar, vector, and tensor operations that are inherent to AI processing. Our design approach of optimized scalar, vector, and tensor acceleration combined with large local shared memory, dedicated power delivery systems, and other hardware acceleration differentiates our solution."

"You know, I reckon if we put a supercharger on this Model T, it'll be as good as the rest."

 

[Getupthere]

Apple’s Next iPhone Could Be The Best iPhone Yet — Forbes

With Apple expected to launch the iPhone SE in the next few months, why could this iPhone be one of the best smartphones to buy in 2025?

apple.news

While Apple is late to generative AI, it continues to innovate with other techniques that contribute to the iPhone experience. You’ll find Apple’s Neural Engine built into the Axx chipsets, which allow for on-device language processing, image recognition, and data processing through machine learning.


When there is so much talk of moving personal data into the cloud or processing (as well as the intense energy demands of generative AI), many will look at Apple’s efforts to keep data on the device and find it a more personable decision.