Data Update Erased Files

I started the Norgate Data Updater (NDU) this morning for the first time in a while.

It said that I needed to update my .NET Framework to v4.7.2. So, I did this and restarted my computer and the NDU.

NDU then displayed a prompt saying that a folder was missing and prompted me to either search for it or to "Restore" it. I chose the Restore option. NDU then said that it needed to update its version and restart, which it did. Then, the data update proceeded without issue.

I then opened Amibroker and right-clicked on the "Maintenance" prompt in the lower right hand corner. The maintenance process proceeded to completion without issue.

Following this, I opened some of my recent layouts and charts but noticed that they seemed to be old (from work that I'd done in 2018).

I then noticed that all of the .AFL files that I have created since February of 2019 are missing; i.e. the files no longer exist in the directories (or any others) that I created/saved them in.

As far as I can tell, the entire past year of work that I've done is gone.

A factor to consider is that I previously had two subscriptions: 1. Futures, 2. U.S. Equities, but let the futures subscription expire because I no longer needed it. During the update/maintenance, it seems that NDU/Amibroker removed these symbols from the DB, which is fine with me... the problem is that it deleted all of my custom code as well.

Is there a way to revert these changes or recover the lost files? Is there a backup created somewhere on my hard-drive prior to each update? I have searched but didn't find anything.

Thank you.

Hi Porcupine,

Note that we do not modify any part of the AmiBroker database folder directly - it is all done through calls to the AmiBroker for adding/modifying/deleting symbols, and we also do not modify any layout files. The only .AFL files we write to are in the Formulas/Norgate Data folder, which only contain the Norgate Data functions.

From what you've suggested, it would appear that your computer has been "rolled back" to a previous date. Did you restore a backup or roll back to a "system restore point"? This is not something Norgate Data would have initiated.

I suggest you contact us directly at support@norgatedata.com so we can look at this further.

Cheers,
Richard.

@porcupine I don't know if it helps (it might not), but you may check AmiBroker --> Formulas --> Drag-drop folder (which by default is hidden) and see if there are any copies of your lost AFL files.

Then look for project files (*.apx) which might also contain copies of some of your codes.

Of course doing a periodical backup of custom files is always the best solution ...

if you can't find any copy of your files elsewhere, try one of those "recovering" utilities such as Recuva (a free one) https://www.ccleaner.com/recuva
Good luck.

I had the same issue when I changed from Norgate premium to the NDU. All of the .afl files in the default Amibroker\Formulas\Custom directory were deleted and when I created new formulas they started numbering again from 1.

I was lucky because I only use the default directory to write and test new things and all of my important code is saved in another directory which is shared between the two PC's I run Amibroker on and backed up to my NAS.

I couldn't find anyone else with the same issue online and at the time I thought I had somehow messed up installing NDU. It also only happened on one of the two PC's I run Amibroker & the NDU on.

I wasn't able to recover the deleted files but I didn't try very hard because I hadn't lost anything which was really important. I hope you have better luck.

I can only reiterate: The Norgate Data Updater app doesn't erase any files in the AmiBroker\Formulas folder.

It only:

  1. Creates AmiBroker\Formulas\Norgate Data
  2. Writes a few Norgate-specific formula files to that folder

If things "disappear" it's most likely due to an incorrect set of permissions on the AmiBroker folder (this typically occurs when somebody copies the entire AmiBroker folder from another computer without first installing AmiBroker) and the way that Windows creates a VirtualStore for Program Files.

Windows upgrades (Windows 7 to Windows 10) also modifies various items in the Program Files folder too and may be a cause.

The lesson from all this is to keep backups. Multiple backups. Test your backups too. Have multiple instances of your trading PC too and keep them in-sync.

Cheers,
Richard.

All, thank you for the suggestions.

I figured out what went wrong. It was something stupid that I had done. It was in no way due to the Norgate Data Updater. I was able to recover my files.

The lesson learned for me is that I've been operating for too long without version control. So, I'm wondering if anybody has any recommendations for either:

  • An IDE/code editor that has integration with GitHub, BitBucket, SVN (or anything like them) as well as having syntax highlighting for AFL?
  • A good workflow/process for creating, versioning, checking in/out, and committing changes to cloud-based repo.

For example, when coding in Python, I use Pycharm, which is integrated with GitHub. So, version control is relatively painless. Whereas in Amibroker, I have to use my own custom naming conventions to keep my file versions organized, and am always at risk of losing everything due to an OS or disk failure.

Thanks.