Hello mr_roboto, thank you for the reply.
What do you mean by "kernel mode access", and why would you need it for this application? All you need is storage. Machine learning should be a regular userspace process.
Kernel mode access, from a programming perspective should be enough to determine what i am talking, but it can depend on the experience of the user. Milage may very, as they say. Sorry your having problems understanding it.
Background:
Kernel mode access: "drivers that run in the same memory space as the operating system kernel, which is the core component of the system that manages resources, processes, and security. This means that kernel mode drivers have direct access to the hardware and can perform faster and more efficiently than user mode drivers."
Ref:
https://support.apple.com/en-gb/guide/security/sec8e454101b/web
Anything that is not run within the kernel, is sandboxed in a different memory space, and apple use programs such as "gatekeeper" to manage this. You can add apps using spctl command, and whats supposed to happen is gatekeeper to prompt you when an app is ran, if have set security level within the macOS recovery environment > startup security utility > reduced security > allow user management of
kernel extensions from identified developed.
The SIP commands can disable protection and the SPCTL command to disable protection, both do not work when running the os off of an external NVME, hence “
kernel mode access is being refused” regardless of the settings the user makes. Again this is M2 mini pro Mac.
And i am not just speaking about machine learning, i am also talking about FOSS running on the mac, such as veracrypt having access to mount and unmount encrypted partitions.
Also your generalising machine learning, as some need drivers ran in the memory space of the kernel. This can also be achieved installed on a virtual machine that has kernel access within its own elastic environment, which is the case for your VPS i believe. But this is software or would need a redirect to hardware resource, which some GPU manufactures provide API access for, which can require an additional licence. With bare metal, in no VM i need local kernel mode access. This can give better access the computer hardware, such as the neural engine.
Also, what would be different about booting from an external? You gave a fairly weird answer before that's hard to make any sense out of.
Explain whats wired about wanting to be in an environment that allowed for replacement of an SSD when it reaches TBW? Once it is dead in the mac mini (soldered on) its dead.
How is this strange to you?
What Is not understood? Help me understand you.
Also, who cares where you boot from, why can't you just tell your software to use whatever storage you want it to use? It's almost inconceivable that it's so badly written that it forces all writes to go only to the boot drive.
Well the point of communicating in this forum was to ask for help and let me know what the experiences are of others. So heck, if you think its inconceivable, why not try out my setup and let me know how you would program this.
Things to note, its not made to be a "
consumer level" application, its made to "
engineer" with. If your looking at it from a user perspective, then that may be why its “
inconceivable", try opening your mind more, not everything is set in stone or only what you see. Again, milage of understanding on what I am attempting to achieve may very, but again, I am happy to hear any ones opinion and feed back on the matter. ^^
A lot of the things you've been saying in this thread lead me to believe that you're in way over your head, have no idea what you're doing, and have been freaking out because you googled things, misunderstood them, and leapt to the wrong conclusions.
Way over my head? I had no idea you thought you where my supervisor rotfl, XD.
And Even if it was, i welcome it so i can find a solution, is that not what a good engineer does, find and develop solutions?
This is not over any ones head, whats the problem, kernel mode is not providing access to known developers, applications on macOS when running off of an external drive, even when setting it up to. How is that over my head?
And who's freaking out here? I am not the one who seems to be triggered by asking a question. I am not to do this?
Also I would echo @leman in saying that if you're training ML models on such a tremendously large data set that you need 2500 TB/month worth of writes, why on earth are you doing that kind of work on a Mac Mini?
To answer why i am doing this on a mac, TOPS performance, the macs "neural engine" that any one can access and use.
How did you ever plan to do all this on its internal drive?
I didn't, hence the problem i have stated many times i am having, how much clearer do i need to be?
Kernel access to when running macos on an external NVME. <-- still not clear?
the normal machine to "acquire" for this kind of work is (if you're only doing it once or twice) not to buy, but rent a big AWS instance for a month or two. One stuffed with a bunch of GPUs to accelerate model training. (This is one of the many reasons why I suspect you and/or your boss have no idea what you're doing.)
Normal for who? Perhaps you? Should we all follow only what you do?
You realise there are massive caveats to doing things only in the cloud? For one I am not a company looking to build a solution at scale, I am a student with limited funds. Rotfl!!
There are many many solutions to a problem, and renting tenants on a VPS instances can have hidden additional charges, navigating pricing on providers like AWS is often a nightmare, and if your doing this within a company, they usually have an accounts manager they work with. I am not a company.. Also more and more companies are seeing the inflationary costs with cloud, and moving back to premises equipment and private cloud as a result. Are you perhaps doing this with a company that has unlimited funds?
The point is to be off line, and maintain Intellectual property, which doing so on a VPS will not cater to.
On data privacy and security, regardless of what the provider tells you. have you read your contracts lately? There is an indemnity clause regarding the safeguarding of data, which often states (depending on provider) that they take no responsibility for your data in transit, and will try their best to protect it, often prompting for you to pay another service fee for backups..
Depending on your location, the recourse will be with a governing body (The office of the information commissioner in the UK), and they will not action unless a large number of people (often more then 1000 in the uk) are affected. So, the individual has a problem if they agree to this, and continue to use these 3rd party systems, rather then come up with a solution.
Also, why would i want to train the VPS providers AI (AWS in your case) with my data? Your response makes no sense!
Again apple have stated the m2 Mac mini pro supports known developer kernel access on macOS off of an external drive. So this should be working. I am still working with them to solve this, but at the same time was asking here if any one has experienced this issues.
I am starting to regret asking the public on a forum, as I seem to be triggering a few people who will not actually reflect on what I have to say with an open mind, refuse to answer my question, attack me with ad-hominems “ freaking out” and then state that their way is the only way….