"Ambae volcano is in an ongoing moderate eruption state", a statement from Vanuatu Meteorology and Geohazards Department said.
Evacuees are being housed in tents, school gyms, and government buildings in neighbouring villages.
"The danger line on the map is not clearly visible on the field, hence many people are afraid", he said.
Since Friday, the volcano has been at its highest alert level - prompting authorities to establish a 7.5 mile exclusion zone.
He said there is no reason for them to evacuate and they've become a "burden" on genuine evacuees and the temporary shelters set up to receive them.
National Disaster Mitigation Agency spokesman Sutopo Purwo Nugroho said hundreds of thousands of face masks will be distributed in Bali as part of government humanitarian assistance that includes thousands of mattresses and blankets.
Indonesia's government has set aside a relief fund of almost $US150m, while the country's disaster agency has dispatched face masks, mattresses, blankets and tents to vulnerable areas.
Five mobile sirens have been installed in the danger zone to warn residents in the event of an eruption.
Exodus from Bali volcano nears 100000 as tremors intensify
Meanwhile, thousands of cows have been left behind by villagers fleeing the volcano.
The volcano last erupted in 1963, killing around 1,100 people.
It says a monitoring center recorded more than 800 earthquakes so far Wednesday and thin smoke has been observed rising 50 meters (160 feet) above the crater. Life went on as normal when ash and gravel was falling on us, until the big lava came out and destroyed everything, ' said 82-year-old Gusti Ayu Wati.
The mountain, which sits about 70 kilometers northeast of Kuta, is among more than 120 active volcanoes in Indonesia.
"Mount Agung is entering a critical phase".
The airport in Bali's capital Denpasar has not been affected, but several countries including Australia and Singapore have issued advisories warning travellers to exercise caution.
There is already a comprehensive plan in place to divert flights to 10 other airports across Indonesia, should ash and volcanic gases interrupt scheduled flight plans.