Sure you can debug your DLL in debugger like Visual Studio or any other, but you have to use version 6.16 or earlier (6.10 or 6.00) official.
These versions can be downloaded from members’ zone (http://www.amibroker.com/members/bin/)
You can have multiple installations of AmiBroker so you can have older one for debugging and newest one for normal use.