Here is the trick: create a relation to self, use Match Conditions to select the fields for duplicates finding. Then add a Summary column to calculate # of records (duplicates). When you have more than one, you have duplicates. Check the settings of the Duplicate Contacts relation for details.
To prevent duplicate entries, you should create a validation rule with the condition [Duplicate Count] is greater than 0. Moreover, you have to add the additional Match Condition Id<>Id to the relation. Due to this match condition the system will allow to modify existing records and save changes. The example is adjusted in the Companies table.
Any support materials, such as spreadsheets and real-life work samples would be very helpful as they would help us better understand your business process.