The error could be caused by the call of "hdbnsutil -initTopology" which creates the system and the initial license. An out of memory error could occur if this is a VM with less than ~5GB of RAM allocated (hdbnsutil tries to allocate 4GB of memory).
You can check if this is the cause of the error, as hdbnsutil will produce a trace dump in the directory "/usr/sap/<SID>/HDB<##>/<host>/trace/".
Check the trace files "nameserver_<host>.*.*.trc" for the error "OUT OF MEMORY".
If this is the case, increase the memory to the VM and restart.